LLMs do not confabulate or hallucinate. They predict the next word in a sequence based on correlations derived from a lot of training data. Sometimes that data happens to be structured enough to statistically generate a word that is fluent and corresponds to what we as readers understand as reality. Sometimes, due to the unclear nature of "truth" and the necessarily incomplete data used to train the model, the word or words that are statistically generated are merely fluent but do not correspond to reality.
The answer to "What are LLMs but humans with extreme amnesia and no central coherence?" is that they are not like humans at all, and only resemble them superficially due to our anthropomorphic tendency to infer a mind upon seeing fluent and roughly correspondent text.
> they are not like humans at all, and only resemble them superficially due to our anthropomorphic tendency to infer a mind upon seeing fluent and roughly correspondent text.
Alternatively, we overestimate human abilities, and imbue our thoughts with deep meaning and significance, simply as a way to feel more important than we actually are. Ultimately, we may rely on statistical prediction quite a bit as well.
Obviously, LLMs aren't at the level of human cognition, but the impressive results they do produce, given their obvious simplicity, hints that intelligence may not be as hard to reproduce in silicon as we have previously assumed.
> Obviously, LLMs aren't at the level of human cognition, but the impressive results they do produce, given their obvious simplicity, hints that intelligence may not be as hard to reproduce in silicon as we have previously assumed.
People said this about calculators as well, or initial chess AI as well. "If computers can do this, then the remaining parts of intelligence is probably not hard to reproduce!".
So we already know many humans massively underestimate the depth of human intelligence. It happened many times before and will continue to happen every time we make computers do anything new.
> People said this about calculators as well, or initial chess AI as well. "If computers can do this, then the remaining parts of intelligence is probably not hard to reproduce!".
People also said that chess computers would never beat the best humans. And then when that happened they said, "well they're just using sheer brute force, they will never produce beautiful chess". And then Alpha Zero came along, and showed that prediction was wrong as well.
Personally, I believe that silicon intelligence will evolve a lot faster than ours did. But regardless, I've yet to hear any convincing argument why it can't happen eventually.
> I've yet to hear any convincing argument why it can't happen eventually.
That is a strawman, basically nobody argues this.
> People also said that chess computers would never beat the best humans
How many did say this? I see way more overly optimistic "humans are stupid" messages than I see "computers will never beat humans" messages. The argument is mostly between "humans are obsolete in a decade at most" versus "we would be lucky to see general intelligence in our lifetime", not that it it is impossible.
I don't know about chess, though earlier computers didn't have the power, so people could've reasonably said that.
But we certainly saw it with Go. Many people said that a computer could beat a person at chess or checkers, because they were simple, limited games, but Go was too complicated, too many possible combinations. A computer could never beat a Go champion. And here we are, with multiple generations of AlphaGo pushing the boundaries. They might use a couple thousand CPUs and a few hundred GPUs, but they can do the impossible thing.
* Using the algorithms they used to beat humans at chess.
I were there during those discussions, the AI optimists argued that since computers could beat humans at chess GAI would soon be here. Then it is reasonable to argue that even a simple game like GO is impossible to solve the way we solved chess, so there needs to be more fundamental breakthroughs before GAI can happen.
That same thing is playing out today, and the AI proponents are still misunderstanding the other sides argument. The argument isn't "computers can never do this", the argument is "the algorithms and methods we use today can't do this, so there is no clear path to intelligence from where we are".
> That is a strawman, basically nobody argues this.
Some people do. Some people believe that intelligence is much more fundamental to the universe, and not capable of being reproduced mechanistically.
But if you grant that it is possible, all we're talking about is when; which is anyone's guess. It could be tomorrow, it could take a million years. I'm not pretending to know, just offering a different perspective than the original comment.
But those "some people" aren't here. What we are discussing is how close LLM are to human reasoning. Many argue LLM are already there, "what evidence do you have that humans aren't essentially just an LLM?" basically. You see such posts in this very thread.
But a large majority of people who say "LLMs are nothing like humans" are not making that argument. If you look through these kind of threads on HN you see maybe one post making that argument among hundreds, that is "basically nobody" to me given how often that strawman is used against people who are skeptical about LLMs potential capabilities.
I don't think LLMs are going to get us to AGI on their own, but it's a massive step: an obviously more fruitful one than Deep Blue or even AlphaZero, and also obviously more useful and disruptive to society.
Some people think we can't get to "real" intelligence through Turing-equivalents. Some think we can just tack a little more on an LLM and be there. It's probably something between, but the past-me who thought AGI was definitely >40 years away is much less convinced of this now.
Could it happen in the next 5 years? Proooooobably not, but is there a significant chance? Yes.
This isn't a response to the OPs point, where they've stated their beliefs and their logic for them.
To it, you've basically said: "well in that case defend the times some other people have been incorrect if you're sure this time".
Why should they? The OP might be wrong, but whether they are or aren't has nothing to do with whether some unrelated, half-remembered third parties were right or wrong when mispredicting something else. You haven't addressed the content of the OP's actual argument at all, nor established a basis on which it may share similarity with other mispredictions.
> but whether they are or aren't has nothing to do with whether some unrelated, half-remembered third parties were right or wrong
Their argument was "Alternatively, we overestimate human abilities". Since he made that argument it is relevant to see if humans overestimated or underestimated human abilities in the past. My argument is that there is a large tendency for smart people and researchers to underestimate human intelligence, to assume it is easy to solve, we have seen that happen many times in the past.
So since smart humans often underestimate human abilities, the "we overestimate human abilities" theory is probably not correct.
Except that isn't true at all: the history of animal research is one of us assigning all human behavior as "uniquely human, requiring great human intelligence" and then discovering it's either replicated in detail in other "lower" animals, or replicated by animals which don't have nearly the apparent cognitive capacity for it[1].
We have an enormous history of vastly over-estimating the intelligence needed for some behaviors to manifest, with the more accurate conclusion being that actually we simply don't understand the nature of intelligence very well.
You suggest we under-estimate animal intelligence, which is the same argument I'm making. The reason we underestimate how hard human intelligence is to solve is that we underestimate the animal part of our reasoning, animals can't play chess or do math or write sentences, so when computers do that people assume they are smart like humans. But that assumption hinges on the animal part, like navigating a forest, identifying plants to eat etc, isn't a fundamental part of intelligence or that it is easy to solve in comparison.
So yeah, underestimating animal intelligence is a large part of it, I agree with you.
To me predicting the next token is obviously not how humans think.
If I ask you to envision a green triangle and a red square next to each other, and then swap the shapes but keep the colors in the same locations, and answer what color is the triangle now, you say the triangle is red, but you do so because you envisioned the triangle swapping places and did the mental steps etc.
An LLM if even answering correctly, is statistically answering based on billions of lines of text + rlhf and all of this, I highly doubt there is a mental model of the world, but rather a large set of constraints in the probabilities which leads to the resulting answer. The reasoning ability is a secondary effect of the probabilities which is why it's hard to make it so every probability is correct for every answer I think.
And regarding OP about hallucinating vs confabulating. To me hallucinating is a fine word for it, because it is filling in a gap or there aren't enough constraints in the model/data/tuning to account for that specific answer that it gave that was incorrect. Hence it "hallucinates" something in the gap. The real power of LLM's is that it seems to accumulate these 'constraints' (generalization), so that with the right model, it should be able to answer more and more prompts correctly, which is kind of amazing.
Confabulation works too but is a little more high level IMO.
I'm not entirely sure if the mental model is somehow a layer deeper than the prediction that humans do. I used to believe it, and still use it as shorthand, but these days I'm not sure it's accurate.
The triangle example doesn't prove it because our predictive model could also say "hey, things don't just change color and shape like that, they need to move". It's similar to how an LLM can be more accurate with math when asked to step through and reason through it's logic - by stepping through the individual steps it can create a larger system.
The thing that made me question if a "mental model" is at the base of human cognition - people who do those memory competitions, the clear winning strategy is the memory palace, or imagining walking through a house where each room is another number - they have to build step by step memory, it's not like an SQL database where they can just SELECT from random. Another one was the insight from GTD that if you remember 7 things to do, you're always repeating those 7 things to yourself to keep them in active memory.
There's a strong argument that the mental model is derivative of a predictive model in the human brain, and we can just appear to have a mental model since we have an internal dialogue that runs so fast in the background that even we rarely recognize it. (anyone who has kept a steady meditation or similar practice should be familiar with it)
Well I can't say any of this for sure but I want to say upfront, I think llm's can in theory do a lot (not sure if most) of the computations a human can do, but it's important to realize, imo, it's not actually stepping through the steps in the way humans are. When we give complicated step by step prompts and so on, it only means it's creating new constraints for what the probability of the next token is (from what exists in the data/model). If the data/model doesn't contain the data needed to produce the desired result, or the data that it was trained does not have examples that can generalize (but not be specific to) the desired result then it can't produce it.
That's the difference between humans and llm's imo. We can generalize any "computation" we have to any other "desired output" we want, by thinking about it, while llm's aren't at least not now, so general that they use low level representations of all the 'objects' we can prompt about. Like humans can reason about the objects and things in our mind almost infinitely and recursively while also retaining all the physical realities and facts of those objects, while an llm is limited in this regard. Doesn't mean it can't in theory, there is some weird generalization going on as far as I can tell, but it feels like it's going to need a lot more data or something to do it.
But humans clearly do that. It's not all we do, but we clearly do that. And given the impressive results that LLMs produce, my guess is it or something like it, will play a significant role in the AGI of the future.
I'd say possibly it is all we do, we just have a lot more hidden inputs. i.e. we're all dealing with inputs from our physical bodies now, and can't not be. There's a whole bit of social advice about dealing with people which is "remember you may be encountering someone on the worst day of their life" (the idea being, 95+% of the time you shouldn't escalate, particularly with strangers).
What I do now, and how I think about things is being affected in ways I know are directly related to blood sugar, fatigue, pain (i.e. my legs are a little uncomfortable in a way yours aren't, but I moved them just now so they feel better - all unique inputs to me, all always there).
But the other problem with the notion is people thinking of the knowledge bounds these systems derive as being "limits" - they're not. The whole point of machine learning is they're not interpolating between known datapoints, but rather extracting the rules which fit those data points. So stay between the points, and sure - it's some type of complex interpolation of seen inputs. But you don't have to stay between those datapoints - you can move of either end and extrapolate new ones entirely.
I would say LLM is at best a precursor for a memory system. But saying we almost have AGI because we have a template for HDDs isn't exactly representative IMHO.
Whether the model for human intelligence is popular or not isn't really relevant. We just care about accuracy. Predicting what comes next (including what will happen if I take some action) is very close to all we do. Models trained on text happen to do surprisingly well because humans often represent predictions of what will come next in words, but it remains to be seen if enough of what it takes to think at a human level is available as text in order to train a human level intelligence purely from text.
Why doesn't passing the Turing Test mean you are sentient is an open question but most agree that it is obvious there is a gap here.
"Predicting text" is just a falacy born from applying function logic to humans. "Well if I simplify the problem to a text window obviously they are the same" as if simplifying doesn't change anything.
The Chinese Room is a good example if you want deeper thought into the distinction between acting and being.
> The Chinese Room is a good example if you want deeper thought into the distinction between acting and being.
It's not a deeper thought at all. It's an appeal to the emotion of humans who want to believe that what we do in our heads somehow produces "understanding", while the procedural steps inside a computer doesn't. But to my mind, it doesn't give a convincing reason for this assertion. You can't prove that what we're doing in our heads, isn't functionally equivalent to what is going on inside some future AGI. There's no such test, just hubris.
As far as the Turing test goes, if you can't tell the difference between a computer and a human, what is the point of even arguing that humans still hold some claim to being the only one in the fight that actually "understand"?
The Chinese room is not saying humans are smart. It is showing the difference between knowledge and action.
Understanding a language is distinct from responding to language prompts.
Unless you are claiming LLM is AGI which is a laughable proposition the fact that people say LLM isn't intelligent isn't impactful on whether AGI isn't intelligent.
There isn't disagreement (outside extremists) that AGI doesn't exist yet. The only disagreement is "when do we know it does".
Not having a test isn't hubris. It instead shows it is a fundamentally hard question. What does intelligence mean?
I don't know exactly how humans think or how LLM's "predict the next token", but your argument is just refuting a high level description of behavior in place of a low level description.
To me at least, it's a bit like saying a car's wheel do not spin when stepping on the pedal, it's the petrol engine that makes the wheel spin. You can replace petrol engine with an electric engine, and the description about the wheel spinning when stepping on the pedal is still technically correct regardless of any lower level explanation.
We don't know how to describe the behavior of LLM's because of how foreign they are to us right now. Hallucinations are confabulations are meant to describe human behavior of couse, so the downside is it might make us anthropomorphize LLM's. However it's the best we could come up with I guess.
If confabulation is a more precise explanation of behavior then I don't see why that's a bad idea.
We do know how to describe their behaviour. It's in terms of plausibility. The central dogma of LLM-as-chatbot-assistant is that at sufficient scale, plausibility converges towards accuracy, and becomes a proxy for utility.
This is not proven, but that is incidental to the deeper issue.
The deep issue is that when a "conversation" kicks off with the LLM being confidently incorrect, the most plausible continuation to take from human literature, and from a good deal of human interaction, is that they continue to be wrong.
I don't think it's that deep of an issue, or at least OpenAI seems to know how to fix it. ChatGPT often tell me when it's wrong and it also often recognizes that it has made a mistake when prompted (e.g. "are you sure?").
Yes, I too have seen ChatGPT issue an apology. Then it gives a worse version of its previous answer, buggier code etc.
And it's because the most plausible continuation of that interaction is now of an ongoing conversation with an apologetic incompetent.
Do not anthropomorphize the LLM. It does not have mental state. It does not have feelings. It does not have a suddenly realised goal of "doing better". The written apology was merely the most plausible text to issue at that point. It will then go on to simulate being wrong again, which I've come to recognise is more plausible than an idiot suddenly becoming a wizard.
In such circumstances, the actual best remedy is to start afresh with a revised prompt.
Most of the people who repeat this have hardly any background in neuroscience... so they actually have no clue at all what the process they're claiming LLMs don't mirror is.
Meanwhile the people who are in neuroscience have posited a Bayesian model to human cognitive function as far back as the 1800s, well before any convenient targets for anthropomorphization existed, and it continued as an area of study until today: https://global.oup.com/academic/product/bayesian-rationality...
Essentially, in typical techie fashion people with barely any understanding of a topic assume to have mastery ahead of people are the forefront of a non-tech field.
_
This a sort of objectification seems equally as rooted in emotion and feelings as the anthropropmization crowd: We don't actually know the answer to the question of if there is some layer of human cognitive function that mirrors LLMs, but it'd cause me great cognitive dissonance/discomfort if the answer is not <insert preferred answer> so I will vehemently fight against any claims to the contrary.
To me the answer is to simply accept we don't know, and take whatever tools let us make progress on working with the LLM. If that's borrowing words like "reasoning" and "understanding" and "confabulate" so be it, it's of no loss to anyone.
You fundamentally do not understand the works you are linking to and it's hilarious that you would use it as a way of saying that neuroscientists think otherwise.
I would note for other readers that the commenter didn't go to college for neuroscience, or any science, based on their previous comments.
You didn't understand what I said, what the link implied, or how they were related, yet you forced a reply.
I hope that your insecurity was patched over at least as long as it took to write the comment: that might partially offset the time anyone else takes to read it.
> they are not like humans at all, and only resemble them superficially
We have no idea how LLMs work on a high level, and we have no idea how the human mind works on a high level. Therefore, such claims are rather overconfident. The fact that the two are dissimilar at the plumbing layer doesn't mean they cannot be alike in how they "really" operate. Either way, we simply do not know, so any such talk is (bad) philosophy, not science.
This is just incredibly wrong. Do you think that LLM's were created out of thin air? A random combination of bytes that we happened to discover that we also happen to be improving/changing?
We understand the training algorithm, but not the network that was trained.
There’s the whole interpretability sub-discipline which has limited but interesting results. Those just indicate even more that we really have no idea how something like GPT-4 works inside its trained model.
Nobody knows how LLMs work on a high level. There's an algorithm that consumes Terabytes of raw text and produces 175 billion parameters, that when used as a statistical model magically reproduce human language. But the conceptual connection between those parameters and what the model can do is a complete mystery.
> The answer to "What are LLMs but humans with extreme amnesia and no central coherence?" is that they are not like humans at all, and only resemble them superficially.
Is there any chance at all that we humans are are also just predicting the next word in the sentence? That the "voice in your head" that so many people report having is itself some kind of LLM that is, to some extent, driving their reasoning?
> Is there any chance at all that we humans are are also just predicting the next word in the sentence
No, humans live in a world and use their intelligence to manipulate the world. When we generate word sequence we usually generate those to communicate information about the world, not to try to parrot what others have said. Humans can use their intelligence to parrot what others say, many do, but we know for sure it is not all humans do.
"Human's don't just parrot what other people say, therefore we must be doing something more than what LLMs do, because an LLM only parrots the words that it is trained on. But an LLM is not truly intelligent because it is not doing what humans are doing—and because the LLM is not intelligent, all it is doing is parroting the training data."
Where in this line of reasoning is it proved that the mechanisms are different?
What if, instead, the conscious intelligence that differentiates humans from other animals is an emergent quality of language learning, specifically? Isn't there evidence that children that don't have access to language—deaf children who are not taught sign language, or feral children, or seriously neglected children—often cannot reason before they are taught language, and suffer from long-term cognitive impairment for having lacked language during their early development? Wouldn't this point to a fundamental linkage between language and human intelligence?
> What if, instead, the conscious intelligence that differentiates humans from other animals is an emergent quality of language learning, specifically
Sure, you can argue that you have solved the human exclusive part of intelligence, that is a possibility. But we have not yet solved the monkey part of intelligence, the part that all mammals possess that lets them act intelligently in this world. Without such animal intelligence I believe it is impossible for the model to not make really stupid mistakes no human would make, because it lacks the intuitive understanding of the world all animals has.
I don't think training on text will ever produce that level of understanding, no matter how hard you try, text just isn't the right medium to build an intuitive understanding of reality like a dog has.
At least I used to think that, but LLMs are basically a very unexpected (to me, at least) counterexample to that theory:
Formulating plausible sounding sentences seems to be possible without having a much deeper world model.
> What if, instead, the conscious intelligence that differentiates humans from other animals is an emergent quality of language learning, specifically?
There’s various linguistic theses claiming similar things (“human minds are wired for language”, i.e. Chomsky’s universal grammar, and conversely “language shapes general cognition”, i.e. Sapir-Whorf).
At least in the light of LLMs (if not long before), I think neither are actually still serious possibilities/useful models of the relationship between language and cognition.
I fail to see how this makes us different from LLMs.
LLMs are a bunch of computation running on silicon in order to predict the next word, and out of that all sorts of behavior arises, including intelligence, insigt,creativity, humor, and confabluation.
Human brains are a bunch of computation running on carbon in order to <<X>>, and out of that all sorts of behavior arises, including intelligence, insigt,creativity, humor, and confabluation.
We don't really know what <<X>> is. Why couldn't <<X>> be prediction as well?
If your mom says "there is milk in the fridge", do you think she believes there is milk in the fridge and wants you to know that, or do you think she just parrots something she heard about milk and fridges on the radio or somewhere with no intent or basis in reality? Not sure what your argument is really.
I have no clue what you're talking about, but there's fundamentally no evidence our thoughts are the decisive events leading to behavior rather than, say, a rationalization of said behavior. You're too bold in your modeling of human perception.
> there's fundamentally no evidence our thoughts are the decisive events leading to behavior rather than, say, a rationalization of said behavior
In fact, there is plenty of evidence for the latter, such as experiments where people whose motor cortex was stimulated electrically made up all sorts of random explanations for why their body moved, including, crucially, that they intended for that motion to happen.
If you don't make that distinction then your statement doesn't make sense "there's fundamentally no evidence our thoughts are the decisive events leading to behavior". You started making this separation here, not me, I just talk about intelligence. If you thought they were the same thing you wouldn't say things like that.
I just took "intelligence" as a metonym for "thought processes". Again, I am unsure what it means to use intelligence if not through our own perception of our own intelligence, nor what utility such a concept would have.
That claim is the basis to basically all other claims, it is fundamental to such a degree that questioning is in the realms of philosophy and not science. If it is false then just about everything humanity believes and knows is false.
Yes, this is the problem. We have to be able to efficiently describe what LLMs do and how they do it, when what they do is superficially familiar, but how is fundamentally alien. We haven't yet developed the necessary language to discuss them on their own terms.
Anthropomorphism and resorting to metaphors like "hallucinate" and "confabulate" are inevitable if you don't want to have to preface every comment with a paragraph of technical discussion. They get the necessary point across which is that the "reality" LLMs construct is not necessarily tethered to actual reality. They're deceptively convincing but can't be trusted.
I fully agree with this perspective. The terminology will change as the field continues to evolve. As long as any anthropomorphizing terms are chosen carefully and are not aggrandizing, it shouldn't be a problem. IIRC "hallucinate" was a term previously used to describe characteristics other network types such as RBMs and had just been carried over to LLMs.
Confabulation in humans seems much like word/sequence completion, autocompleting from gist information without reality monitoring, executive control, etc. Likewise, introducing output monitoring that can identify signals of recall failure, and executive control (stop signals), probably could help LLMs too.
"They predict the next word in a sequence based on correlations derived from a lot of training data."
This misinformation really needs to finally die.
They are trained based on correlations of next token prediction. But that does not mean that the resulting neural network is only doing that.
When Harvard and MIT researchers fed Othello moves into GPT, it was training based on the correlations of legal Othello moves.
But the version of the NN that performed the best at that task had created within that network dedicated structure representing a legal Othello board and tracking the state of moves - neither of which were things explicitly in the training data or goals the training was directly measuring.
You effectively came to exist by a training process of surviving to reproduce. But it wouldn't exactly be accurate to say that as complexity increased, the only thing you remain capable of is surviving to reproduce, even if many of your emergent capabilities happen to improve the success of being able to do so.
Yeah, i think that neural networks learn because they can represent if/else relation (via step function (activation)) and that makes them learn good heuristics at least.
I wonder if someone can prove that NN can generate arbitrary programs because as we know 2 layer NNs can approxomate any calculation. Given some state and how easy is to create turing complete machine i think they already might have it.
I speculate that the trick is that these programs sometimes dont evaluate in time when response is short. That’s why LLMs should be given way to adjust output and something like stop signal. Adjusting output can be done using DiffusER approach but not sure how to handle stop signal. I guess we could try to add fake input tokens for silience/thinking.
This is a beautiful summary of what happens in reality with these statistical algorithms applied to digitized representations of human language.
But as you can see in the comments, people simply do not want to internalize the mundane truth. Confabulating fake AGI narratives and hallucinating AI supremacy is more fun (also more profitable).
That's exactly it, and that is what is so hard to get other people to understand. It's a random word generator. It can't 'hallucinate' or 'confabulate' anymore than dice can. It cannot possibly be wrong or incorrect.
Those of us who've played D&D or something similar are familiar with rolling dice, say a 10-sided die and a 6-sided die, and cross-referencing the results against a table to generate random monster encounters, treasure, etc.
If you roll a die to determine how many gold pieces an orc is carrying, and you roll a 5, that is the result. You don't claim that the dice was hallucinating or confabulating or got the facts wrong or whatever. It's a random generator, and it generated a random number, and that result is truthfully the result it generated.
LLMs are also random generators. They're not databases of facts, they're not mathematical calculators. They are like the D&D random generation tables, only developed deeply enough such that they can string together randomly generated words into coherent sentences and paragraphs.
That sentence/paragraph is just as true as the result of rolling some dice. If you roll an 11, then you rolled an 11. The dice didn't hallucinate it, confabulate it, lie about it, or get it wrong. The true result is that you rolled an 11. If you wanted to roll a 12, too bad. You were unlucky. The dice weren't somehow magically 'wrong'.
And an LLM is not somehow magically wrong or incorrect or hallucinating or even confabulating when it gives you an answer that isn't what you wanted. It's randomly generating text. And its answer is every bit as correct as the result of rolling dice or drawing a card.
> They are like the D&D random generation tables, only developed deeply enough such that they can string together randomly generated words into coherent sentences and paragraphs.
Here’s an example where ChatGPT wrote code that solves the problem and then makes a bunch of statements that are all true. I can’t imagine any of those statements were even in the training data.
I don’t see how you can say this is the same as drawing a card out of a deck. You should stop trying to get people to understand that. It’s not true.
The answer to "What are LLMs but humans with extreme amnesia and no central coherence?" is that they are not like humans at all, and only resemble them superficially due to our anthropomorphic tendency to infer a mind upon seeing fluent and roughly correspondent text.