The advancement with these LLMs lies in the fact that they can effectively learn to recognize patterns within “large-ish” input text sequences and probabilistically generate a likely next word given those patterns.
It’s a genuine advancement. However it is still just pattern matching. And describing anything it’s doing as “behavior” is a real stretch given that it is a feed-forward network that does not incorporate any notion of agency, memory, or deliberation into its processing.
You are comparing systems that generated completion text based on statistics and correlation with a system that now models actual complex functional relationships between millions of concepts, not just series of letters or words.
The difference is staggering.
It comes about because of the insane level of computational iterations (that are not required for normal statistical completion) mapping vast numbers of terabytes of data into a set of parameters constrained to work together in a way (layers of alternating linear combinations followed by non-linear compressions) that requires functional relationships to be learned in order to compress the information enough to work.
It is a profound difference both in methodology and results.
It's modeling patterns found across the massive corpus of textual training input it has seen -- not the true concepts related by the words as humans understand them. If you don't believe me then ask ChatGPT some bespoke geometry-related brain teasers and see how far it gets.
I want to be clear that the successful scale-up of this training and inference methodology is nonetheless a massive achievement -- but it is inherently limited by the nature of its construction and is in no way indicative of a system that exudes agency or deliberative thought, nor one that "understands" or models the world as a human would.
> [...] no way indicative of a system that exudes agency or deliberative thought, nor one that "understands" or models the world as a human would.
Certainly not - its architecture doesn't model ours. But it has taken a huge step forward in our direction in terms of capabilities, from early to late 2022.
As its reasoning gets better, simply a conversation with itself could become a kind of deliberative thought.
Also, as more data modalities are combined, text with video and audio, human generated and recordings of the natural world, etc., more systematic inclusion of math, its intuition about solving bespoke geometry problems, and other kinds of problems, are likely to improve.
Framing a problem is a lot of the solving of a problem. And we frame geometry with a sensory driven understanding of geometry that the current ChatGPT isn't being given.
the visual cortex in your brain is also "just a pattern matching" system. guess it's not very impressive by your standard.
This[1] isn't my example (it's from another HN user), but if you work as a programmer and you're not absolutely jaw on the floor astonished by this example then I don't know what to say.
Explaining[2] the emergent behaviour is literally cutting edge research. Hand waving this behaviour away as just "probabilistically generating a likely next word" is ignorant.
It's amazing in similar ways to Conway's Game of Life.
I'm arguing against the notion that these LLMs exhibit "emergent behaviour" as you stated. I don't believe they do, as the term is commonly understood. Emergent behavior usually implies the exhibition of some kind of complexity from a fundamentally simple system. But these LLMs are not fundamentally simple, when considered together with the vast corpus of training data to which they are inextricably linked.
The emergent behavior of Conway's Game of Life arises purely out of the simple rules upon which the simulation proceeds -- a fundamental difference.
emergent behavior in this context is defined as: "emergent abilities, which we define as abilities that are not present in small models but are present in larger models"
>The emergent behavior of Conway's Game of Life arises purely out of the simple rules upon which the simulation proceeds -- a fundamental difference.
> emergent behavior in this context is defined as: "emergent abilities, which we define as abilities that are not present in small models but are present in larger models"
Then I don't know why you brought up Game of Life because it obviously has nothing to do with this alternative definition of emergent behavior.
> this is a meaningless distinction.
It's meaningful with respect to the claim that LLMs exhibit emergent behavior in the same way in which Game of Life does.
1. Item 3: The ocean is full of floating objects, and it would be hard to see the duck among them?
2. Item 2: is structured as non sequitur, takes a long time because there are many hazards?
I am impressed that you find it impressive. It is plausible-sounding, and I find that disturbing, but it is not useful (and the text prediction paradigm seems a dead end in terms of formulating anything more than plausible sounding)
Technically, it's not pattern matching. It's estimating conditional probabilities and sampling from them (and under the hood, building blocks like QKV attention aka probabilistic hashmap and the optimization used decide what it does anyway, ignoring any theory behind it).
The anthropomorphism is indeed exactly why this is a big problem. If the user thinks the responses are coming from an intelligent agent tasked with being helpful, but in reality are generated from a text completion model prone to mimicking adversarial or deceptive conversations, then damaging outcomes can result.
I don’t know or care what Chinese internal media is reporting — this is a significant story in the U.S. that is getting less play than it should. It’s an ecological disaster with potentially significant health concerns, and likely involves corporate negligence as well as a failure of governance by both political parties who recently forced rail workers back to work despite complaints related to possible causes of this derailment (and others).
The spy/balloon thing is way less significant and has been ongoing by both sides for years or even decades if my understanding is correct.
> It’s an ecological disaster with potentially significant health concerns
Counterpoint, we deal with hazardous chemical spills all the time, especially when you factor in all of the semi crashes that don't make headlines. There is a very robust set of clean up procedures set aside for these chemicals ahead of time, and they are being followed. What we are watching now are the automatic precautions of our system kicking in as intended - like taking soil and water samples and testing air quality.
The system is working as intended - we only think it's a disaster because we are being told it is on social media. And often the evidence of the system working is being used to argue it's not.
Outsider efforts that seek to undermine these efforts or make big deals of the mundane should be taken with a grain of salt for the same reason ballot recounts shouldn't be proof that our election process is broken.
You must live on another planet if you think we have a good record on cleaning up hazardous spills, in an era where oil wells that can't be plugged continue to bleed into our oceans. Most cleanup operations are matters of triage, we almost never are able to undo the damage, in the best case scenarios, ecosystems are able to recover over time, but in many places, ecological damage might as well be permanent. Entropy is not to be trifled with.
I mean, I live near the Hanford nuclear site. Despite the unseen horrors this patch of land has seen, people continue to grow food in the area and live to ripe old ages.
For sure, I don't really know how much continued damage the gulf spill might have caused. But I have been swimming in the Caribbean since the spill and didn't particularly worry about it anymore at the time.
As far as I am concerned, if the Chinese state is pushing a grassroots recognition of this event that could lead to accountability and improved safety practices, then they are doing us a public service.
Lowering rates will give a boost to the stock market in the short to medium term. In the long term, returns will settle closer to the interest rate in question (plus a risk premium). See: Japan over the last couple of decades.
It is kind of interesting how this was the core observation of Keynes. It isn't capital that dominates the markets it is money.
According to classical economics people either spend or save, there can be no such thing as indecisiveness or paralysis. What this means is that a too low interest rate would immediately cause inflation and therefore result in a higher nominal interest rate because the rate of capital formation is too slow.
In practice we haven't observed any capital shortages that weren't the result of a one off event. The interest rate appears to be the only barrier and the return on capital followed it in countries excluding the USA.
There isn’t much information available on how its indexing works internally, whether tag existence / tag values can be indexed, etc. So it’s hard to judge the extent to which this could be a replacement for Postgres with PostGIS.
Indexing in GeoDesk GOL files is predominantly spatial (a quadtree-like structure combined with r-trees), with probabilistic indexing for the most frequent keys (`highway`, `building`, `amenity`, etc.). This means queries with a spatial dimension execute very quickly (bounding-box queries have a typical throughput of tens of millions of features per second on a simple dual-core notebook). GeoDesk is less suitable for non-spatial queries (e.g. "Where in the world is the Louvre?" may take several seconds on a planet-size database), but we'll consider enhanced indexing for these scenarios if there are enough real-world use cases to warrant it.
I think it’s a bit of a point of no return for both sides. Musk wanted all chips on the table, got them, and it turns out his hand isn’t as strong as he thought. There is no going back to status quo after this email stunt, it literally made clear that the majority of employees are just sticking around until they find something else, and would even leave sooner if at all pushed to. This is a sinking ship.
It’s a genuine advancement. However it is still just pattern matching. And describing anything it’s doing as “behavior” is a real stretch given that it is a feed-forward network that does not incorporate any notion of agency, memory, or deliberation into its processing.