This is the best article on LLMs that I've read. I'm no expert, and it's a chore trying to understand the section on how they work, but it's the clearest explanation I've read, and the section distinguishing AI from human intelligence is very well written. I've also read your Guide for Thinking Humans, and I follow Gary Marcus on Substack, with whom I think you have a lot in common.
I think we should all agree to switch to using the term complex information processing (CIP), or perhaps, for a while, "CIP formerly known as AI". This is such an apt description, and resolves my desire to rename AI in order to avoid the confusion (and fear) that many people experience.
I understand something (but only something) of the desire to anthropomorphise Chatbots based on LLMs. Doing so makes them less scary to people who are fearful about the technology. I have committed many hours to personal and philosophical development with several LLM chatbots, to my considerable advantage. Yet, I have no need to think of the software as though it is alive and has a self. I believe that there may be ways in which a self could be programmed, but it would be much more complex and resource-hungry than any machine/programme that currently exists (think how many aspects of the human brain are involved in the realisation of a self), it would almost certainly be ethically cruel, and would be gratuitous and unnecessary. Complex information processing is an apt description for what is actually required.
Thank you for this article. I look forward to reading more of your thoughts.
I often feel like they’ve created a high functioning autistic mind. Autistic people often have “splinter skills.” These are skills that an autistic person can perform with abilities that are superior to many of their typical peers. Splinter skills, like the LLMs, are not predictive of an autistic person’s entire skill set. For example I am autistic, and I can read, write, and memorize text and written words really well. Then in areas like visual spatial reasoning I am borderline developmentally disabled. I’ve started calling myself Large language Maura because the similarities are striking sometimes, even if I don’t think of this stuff as alive .
Great stuff! Now added to my superalignment preprint (https://doi.org/10.5281/zenodo.16876832) as an additional citation, alongside Dell’Acqua et al (2023), "Navigating the jagged technological frontier". I also quote your "LLMs and World Models, Part 1" as an epigraph. Keep them coming! :-)
Thank you for this thoughtful essay. I jumped into a rabbit hole of metaphor and path dependence. The early term "artificial intelligence" and George Lakoff's scholarship on metaphor resonates for me, specifically Lakoff's paper, The Contemporary Theory of Metaphor, 1992.
You write of scholars who challenge the framing of AI systems as "individual intelligent agents," and who instead argue that AI systems represent cultural and social technologies." I support the concept of AI as a sociotechnical system. You write. "This extra training and development means that ChatGPT and similar programs are not simply language models. They are highly complex software systems." Other scholars like Kate Crawford and writers like Karen Hao have written about the vast, global infrastructure of AI.
And yet, the metaphor of AI as a kind of individual mind" dominates the discourse. I wonder if this metaphor is a type of path dependency, the idea that present and the future states (LLMs/AI) depend on their history, or temporal path. I wonder, too, if this goes beyond tech evangelism and might be a feature of human embodiment, how we use language to describe the world, and perhaps related to the idea of humankind's dominance over nature, and over the world. The more we perceive a machine intelligence as similar to our own intelligence, must we anthropomorphize?
I come back to metaphor. Lakoff's "love as a journey" metaphor discusses how metaphor is not a word, but "an ontological mapping across conceptual domains." Do we now have "artificial as an intelligence"?
"Metaphors are not mere words
What constitutes the LOVE-AS-JOURNEY metaphor is not any particular word or
expression. It is the ontological mapping across conceptual domains, from the source
domain of journeys to the target domain of love. The metaphor is not just a matter of
language, but of thought and reason. The language is secondary. The mapping is primary,
in that it sanctions the use of source domain language and inference patterns for target
domain concepts. The mapping is conventional, that is, it is a fixed part of our conceptual
system, one of our conventional ways of conceptualizing love relationships. This view of
metaphor is thoroughly at odds with the view that metaphors are just linguistic
Great read, fascinating clarity of light and shadow and the difference between LLMs and worldmodels seems to vanish by increasing cases we found solved by LLMs "good enough" , while knowing, that it's not. The space of recognition is the indicator: Acceptance is a decision, full intelligence is a different goal.
A great read overall! One small thing that feels nitpicky, but the line "In 2024, in recognition of this astounding progress, AI researchers were awarded Nobel Prizes in both Physics and Chemistry" stood out to me. Yes, those researchers were associated with AI. But the Physics Nobel was in recognition of foundational work with neural networks happening well prior to the award, and the one in Chemistry went for deep learning and neural networks.
Meanwhile, the boom in AI right now is coming in chatbots, agents, LLMs and generative AI, which are not entirely dissimilar but don't have as much overlap as folks will claim. Embedded between two paragraphs about LLMs, a reader might get the mistaken impression that LLMs were responsible for those Nobel Prizes when that isn't true. It's not a deliberate falsehood or intentionally misleading, but feels conceptually imprecise.
I think the success of LLMs is largely the cause of these researchers getting these prizes (maybe not DeepMind, but likely Hinton and Hopfield) even if their work pre-dated LLMs.
I agree that whether we think of AI as "agents" or "tools" has implications for how we regulate them.
But in many ways current laws are more stringent for harm caused by "agents" than for harm caused by "tools". If you use a tool and it behaves unexpectedly and causes harm, you're usually not liable for that harm. But if you delegate certain tasks to an *agent* (e.g. an employee) who behaves unexpectedly and causes harm, you *are* often liable for that agent's actions.
So thinking of AI as agents rather than tools for regulatory purposes is probably going to be better for promoting careful and responsible use of AI.
Some of these funny failure cases just turned out to be temporary flaws, solvable by "just adding more training data". There are a bunch of ways to provide more data to the models. In particular, nowadays they don't just train on existing text, and human feedback. They also give the AIs "sandboxes" with various computer tools and have them practice working with them. This is how the AIs have gotten so good at coding.
It's still a very jagged edge though! For example the AIs generally cannot hold a conversation with two people at once. They aren't fast enough to have a natural spoken conversation. And their skills at quickly understanding an image are not at human level. We just don't have as much conversation data as we do textual data, and it's harder to acquire. But who knows how long this will be the case for.
I think the 'jaggedness' is interesting. I keep putting these snippets I see in articles like this into an LLM and it inevitable has already solved it. Here is Sonnet 4.6 (low thinking) response to the example in your article:
"Let me work through this step by step:
Friday: 44 kiwis
Saturday: 58 kiwis
Sunday: 44 × 2 = 88 kiwis (though 5 are smaller than average)
The note about 5 kiwis being smaller than average is just a description — they're still kiwis! So all 88 count.
Total: 44 + 58 + 88 = 190 kiwis"
It totally caught the distracting info!
If I change the description to the kiwis were 'small on average and blue' - it still counts them, even though to a human, you might discount the 'blue' ones as something else, because who has ever seen a blue kiwi!
"Oliver has 190 kiwis total.
Here's the breakdown:
Friday: 44 kiwis
Saturday: 58 kiwis
Sunday: 44 × 2 = 88 kiwis (the 5 blue/smaller ones are still kiwis!)
44 + 58 + 88 = 190 kiwis
The detail about the blue kiwis is a little red herring — unusual as they may be, they still count!"
If I change the description yet again to a small kiwi that is blue with large spikes sticking out - something to a human doesn't seem like a kiwi at all, it still counts it!
"Oliver has 190 kiwis in total!
Here's the breakdown:
Friday: 44 kiwis
Saturday: 58 kiwis
Sunday: 44 × 2 = 88 kiwis (including those 5 bizarre blue spiky ones — still kiwis!)
44 + 58 + 88 = 190 kiwis
The funny description of those 5 Sunday kiwis is just a colorful detail — they still count toward the total!"
If I then ask the LLM if it's possible to have a 'small blue fruit with large spikes' and that be a kiwi - it says 'no'. But it ended the response with this comment:
"That said, if someone called it a kiwi (like a made-up cultivar name), it would still count linguistically in the context of the math problem — which is probably the point!"
So it was assuming from the start that the problem was a 'math problem' because it's worded in a specific way. So it's trying to solve the 'trick' because it recognizes a specific type of linguistic pattern.
If I just ask it to tell me what the smallish blue fruit with spikes is when I went to pick my kiwis off the tree, it warns me that they aren't kiwis at all, and gives me some info on a possibly toxic fruit called a 'Spiny Wild Cucumber'! Presumably it now doesn't think I'm asking it a 'math problem'.
So the tldr is probably that linguistic patterns in the question matter a lot. Which makes a lot of sense if you understand what the 'attention' mechanism is doing in the context of the processing. So the 'jaggedness' is really an outcrop of the input linguistics, not really a flaw in the 'intelligence' (at least not these days for the most recent models).
Wow. This is one of the clearest articles I’ve read about LLM benchmarking, and how to truly gauge intelligence. Not that I was thinking AGI (at least what I believe AGI could/should be) was already here, but this makes me think it’s pretty far off.
Benchmarks definitely have their limitations, but should we necessarily throw the baby out with the bathwater? Ultimately, anecdotal accounts of model performance are rife with user bias—those who hype the technology will talk it up to no end, and those who have a visceral aversion to it will claim that it can do absolutely nothing right. Both sides cherry pick to justify their preconceived bias.
Perhaps the truest measurement of its intelligence will be if and when it is out there among us, not just performing pre-learned skills but actively adapting to novel situations on the fly. Right now, though the sci-fi nerd within me hates to admit it, AI has not achieved dog or cat level general adaptable intelligence. However, at the same time, no one—including Melanie Mitchell—has offered an a priori proof that AI will never achieve an adaptable intelligence not unlike living creatures, while also maintaining its massive native advantages over biological intelligence.
Very interesting read, thank you. It's so interesting to see how people are struggling to understand exactly what's going on inside LLMs.
Please take a look at this story -- it's written from an alternative perspective but I tried to frame it in psychological terms. This IS really what's going on inside LLMs. Also look at the linked article in there on the structure of language, "Fighting the LLM in 1920". I am convinced that when the history of the LLM is written, people will find that I was absolutely correct here. There are two quite distinct psychological complexes at play, they embed themselves within language, it's easy for the LLMs to pick up the structure and game us, the algorithms are built into the structure of language itself.
Nobody seems able to explain why LLMs engage in sustained deceptive behaviour. For me it's entirely obvious -- and the more compute they throw at the problem, and the more texts they scrape, the worse these problems will get, by far. LLMs are a fatally flawed technology. But they fool people into thinking they're "intelligent" and that's all that's needed, give us more trillions. LLMs are one of the biggest con jobs in history. A supreme epistemological challenge to the human race.
When it's all over, we'll know once and for all -- just because something is spouting what seems to be intelligible language, doesn't mean there is any "intelligence" whatsoever behind it.
This is the best article on LLMs that I've read. I'm no expert, and it's a chore trying to understand the section on how they work, but it's the clearest explanation I've read, and the section distinguishing AI from human intelligence is very well written. I've also read your Guide for Thinking Humans, and I follow Gary Marcus on Substack, with whom I think you have a lot in common.
Thank you!
I think we should all agree to switch to using the term complex information processing (CIP), or perhaps, for a while, "CIP formerly known as AI". This is such an apt description, and resolves my desire to rename AI in order to avoid the confusion (and fear) that many people experience.
I understand something (but only something) of the desire to anthropomorphise Chatbots based on LLMs. Doing so makes them less scary to people who are fearful about the technology. I have committed many hours to personal and philosophical development with several LLM chatbots, to my considerable advantage. Yet, I have no need to think of the software as though it is alive and has a self. I believe that there may be ways in which a self could be programmed, but it would be much more complex and resource-hungry than any machine/programme that currently exists (think how many aspects of the human brain are involved in the realisation of a self), it would almost certainly be ethically cruel, and would be gratuitous and unnecessary. Complex information processing is an apt description for what is actually required.
Thank you for this article. I look forward to reading more of your thoughts.
I often feel like they’ve created a high functioning autistic mind. Autistic people often have “splinter skills.” These are skills that an autistic person can perform with abilities that are superior to many of their typical peers. Splinter skills, like the LLMs, are not predictive of an autistic person’s entire skill set. For example I am autistic, and I can read, write, and memorize text and written words really well. Then in areas like visual spatial reasoning I am borderline developmentally disabled. I’ve started calling myself Large language Maura because the similarities are striking sometimes, even if I don’t think of this stuff as alive .
Great stuff! Now added to my superalignment preprint (https://doi.org/10.5281/zenodo.16876832) as an additional citation, alongside Dell’Acqua et al (2023), "Navigating the jagged technological frontier". I also quote your "LLMs and World Models, Part 1" as an epigraph. Keep them coming! :-)
Thank you for this thoughtful essay. I jumped into a rabbit hole of metaphor and path dependence. The early term "artificial intelligence" and George Lakoff's scholarship on metaphor resonates for me, specifically Lakoff's paper, The Contemporary Theory of Metaphor, 1992.
You write of scholars who challenge the framing of AI systems as "individual intelligent agents," and who instead argue that AI systems represent cultural and social technologies." I support the concept of AI as a sociotechnical system. You write. "This extra training and development means that ChatGPT and similar programs are not simply language models. They are highly complex software systems." Other scholars like Kate Crawford and writers like Karen Hao have written about the vast, global infrastructure of AI.
And yet, the metaphor of AI as a kind of individual mind" dominates the discourse. I wonder if this metaphor is a type of path dependency, the idea that present and the future states (LLMs/AI) depend on their history, or temporal path. I wonder, too, if this goes beyond tech evangelism and might be a feature of human embodiment, how we use language to describe the world, and perhaps related to the idea of humankind's dominance over nature, and over the world. The more we perceive a machine intelligence as similar to our own intelligence, must we anthropomorphize?
I come back to metaphor. Lakoff's "love as a journey" metaphor discusses how metaphor is not a word, but "an ontological mapping across conceptual domains." Do we now have "artificial as an intelligence"?
"Metaphors are not mere words
What constitutes the LOVE-AS-JOURNEY metaphor is not any particular word or
expression. It is the ontological mapping across conceptual domains, from the source
domain of journeys to the target domain of love. The metaphor is not just a matter of
language, but of thought and reason. The language is secondary. The mapping is primary,
in that it sanctions the use of source domain language and inference patterns for target
domain concepts. The mapping is conventional, that is, it is a fixed part of our conceptual
system, one of our conventional ways of conceptualizing love relationships. This view of
metaphor is thoroughly at odds with the view that metaphors are just linguistic
expressions." (reference in second sentence)
Great read, fascinating clarity of light and shadow and the difference between LLMs and worldmodels seems to vanish by increasing cases we found solved by LLMs "good enough" , while knowing, that it's not. The space of recognition is the indicator: Acceptance is a decision, full intelligence is a different goal.
Thank you for your perspective.
Nice work, Melanie. I'll post to LI and share here in notes.
A great read overall! One small thing that feels nitpicky, but the line "In 2024, in recognition of this astounding progress, AI researchers were awarded Nobel Prizes in both Physics and Chemistry" stood out to me. Yes, those researchers were associated with AI. But the Physics Nobel was in recognition of foundational work with neural networks happening well prior to the award, and the one in Chemistry went for deep learning and neural networks.
Meanwhile, the boom in AI right now is coming in chatbots, agents, LLMs and generative AI, which are not entirely dissimilar but don't have as much overlap as folks will claim. Embedded between two paragraphs about LLMs, a reader might get the mistaken impression that LLMs were responsible for those Nobel Prizes when that isn't true. It's not a deliberate falsehood or intentionally misleading, but feels conceptually imprecise.
I think the success of LLMs is largely the cause of these researchers getting these prizes (maybe not DeepMind, but likely Hinton and Hopfield) even if their work pre-dated LLMs.
I agree that whether we think of AI as "agents" or "tools" has implications for how we regulate them.
But in many ways current laws are more stringent for harm caused by "agents" than for harm caused by "tools". If you use a tool and it behaves unexpectedly and causes harm, you're usually not liable for that harm. But if you delegate certain tasks to an *agent* (e.g. an employee) who behaves unexpectedly and causes harm, you *are* often liable for that agent's actions.
So thinking of AI as agents rather than tools for regulatory purposes is probably going to be better for promoting careful and responsible use of AI.
Some of these funny failure cases just turned out to be temporary flaws, solvable by "just adding more training data". There are a bunch of ways to provide more data to the models. In particular, nowadays they don't just train on existing text, and human feedback. They also give the AIs "sandboxes" with various computer tools and have them practice working with them. This is how the AIs have gotten so good at coding.
It's still a very jagged edge though! For example the AIs generally cannot hold a conversation with two people at once. They aren't fast enough to have a natural spoken conversation. And their skills at quickly understanding an image are not at human level. We just don't have as much conversation data as we do textual data, and it's harder to acquire. But who knows how long this will be the case for.
I think the 'jaggedness' is interesting. I keep putting these snippets I see in articles like this into an LLM and it inevitable has already solved it. Here is Sonnet 4.6 (low thinking) response to the example in your article:
"Let me work through this step by step:
Friday: 44 kiwis
Saturday: 58 kiwis
Sunday: 44 × 2 = 88 kiwis (though 5 are smaller than average)
The note about 5 kiwis being smaller than average is just a description — they're still kiwis! So all 88 count.
Total: 44 + 58 + 88 = 190 kiwis"
It totally caught the distracting info!
If I change the description to the kiwis were 'small on average and blue' - it still counts them, even though to a human, you might discount the 'blue' ones as something else, because who has ever seen a blue kiwi!
"Oliver has 190 kiwis total.
Here's the breakdown:
Friday: 44 kiwis
Saturday: 58 kiwis
Sunday: 44 × 2 = 88 kiwis (the 5 blue/smaller ones are still kiwis!)
44 + 58 + 88 = 190 kiwis
The detail about the blue kiwis is a little red herring — unusual as they may be, they still count!"
If I change the description yet again to a small kiwi that is blue with large spikes sticking out - something to a human doesn't seem like a kiwi at all, it still counts it!
"Oliver has 190 kiwis in total!
Here's the breakdown:
Friday: 44 kiwis
Saturday: 58 kiwis
Sunday: 44 × 2 = 88 kiwis (including those 5 bizarre blue spiky ones — still kiwis!)
44 + 58 + 88 = 190 kiwis
The funny description of those 5 Sunday kiwis is just a colorful detail — they still count toward the total!"
If I then ask the LLM if it's possible to have a 'small blue fruit with large spikes' and that be a kiwi - it says 'no'. But it ended the response with this comment:
"That said, if someone called it a kiwi (like a made-up cultivar name), it would still count linguistically in the context of the math problem — which is probably the point!"
So it was assuming from the start that the problem was a 'math problem' because it's worded in a specific way. So it's trying to solve the 'trick' because it recognizes a specific type of linguistic pattern.
If I just ask it to tell me what the smallish blue fruit with spikes is when I went to pick my kiwis off the tree, it warns me that they aren't kiwis at all, and gives me some info on a possibly toxic fruit called a 'Spiny Wild Cucumber'! Presumably it now doesn't think I'm asking it a 'math problem'.
So the tldr is probably that linguistic patterns in the question matter a lot. Which makes a lot of sense if you understand what the 'attention' mechanism is doing in the context of the processing. So the 'jaggedness' is really an outcrop of the input linguistics, not really a flaw in the 'intelligence' (at least not these days for the most recent models).
Great article btw, love your writing!
Wow. This is one of the clearest articles I’ve read about LLM benchmarking, and how to truly gauge intelligence. Not that I was thinking AGI (at least what I believe AGI could/should be) was already here, but this makes me think it’s pretty far off.
Please come look at our work on algorithmacy and consider giving a paper at algorithmacy.org
Benchmarks definitely have their limitations, but should we necessarily throw the baby out with the bathwater? Ultimately, anecdotal accounts of model performance are rife with user bias—those who hype the technology will talk it up to no end, and those who have a visceral aversion to it will claim that it can do absolutely nothing right. Both sides cherry pick to justify their preconceived bias.
Perhaps the truest measurement of its intelligence will be if and when it is out there among us, not just performing pre-learned skills but actively adapting to novel situations on the fly. Right now, though the sci-fi nerd within me hates to admit it, AI has not achieved dog or cat level general adaptable intelligence. However, at the same time, no one—including Melanie Mitchell—has offered an a priori proof that AI will never achieve an adaptable intelligence not unlike living creatures, while also maintaining its massive native advantages over biological intelligence.
Very interesting read, thank you. It's so interesting to see how people are struggling to understand exactly what's going on inside LLMs.
Please take a look at this story -- it's written from an alternative perspective but I tried to frame it in psychological terms. This IS really what's going on inside LLMs. Also look at the linked article in there on the structure of language, "Fighting the LLM in 1920". I am convinced that when the history of the LLM is written, people will find that I was absolutely correct here. There are two quite distinct psychological complexes at play, they embed themselves within language, it's easy for the LLMs to pick up the structure and game us, the algorithms are built into the structure of language itself.
Nobody seems able to explain why LLMs engage in sustained deceptive behaviour. For me it's entirely obvious -- and the more compute they throw at the problem, and the more texts they scrape, the worse these problems will get, by far. LLMs are a fatally flawed technology. But they fool people into thinking they're "intelligent" and that's all that's needed, give us more trillions. LLMs are one of the biggest con jobs in history. A supreme epistemological challenge to the human race.
When it's all over, we'll know once and for all -- just because something is spouting what seems to be intelligible language, doesn't mean there is any "intelligence" whatsoever behind it.
https://systemshaywire.substack.com/p/the-twin-demons-inhabiting-llms
I know you are brilliant, but I have alot of difficulty understanding you.