Since we know what LLMs do internally (pattern matching on sequences of undefined tokens), why would anyone expect LLMs to act in any way other than that described here?
(OK, my _opinion_ here is that LLMs are nothing other than an inane party trick*, so this result is exactly what _I_ expect. But, in all seriousness, the whole LLM thing seems wildly overblown.)
From an AI safety standpoint, understanding that LLMs generate exactly and only random recombinations of their training data, would seem to be important. LLMs are useful for generating boilerplate text that can be edited to match the paper being written, can generate code that can be debugged to do the processing required. But can never be trusted to say things that are true. (Again since they have no model of, do no processing to deal with, whatever it is that "true" means.)
*: Party trick seems to be exactly the right technical term for what LLMs do. Generate random text and let the user figure out if it means something. We say "oops, it halucinated" when it says something stupid, and go "koooooooooooooool" when there's a reasonable interpretation of the output. This is exactly what card/tea/palm readers do.
> Since we know what LLMs do internally (pattern matching on sequences of undefined tokens), why would anyone expect LLMs to act in any way other than that described here?
One possibility is if one considers that "we know what LLMs do internally" is not actually an exhaustively true statement.
> OK, my _opinion_ here is that LLMs are nothing other than an inane party trick*, so this result is exactly what _I_ expect. But, in all seriousness, the whole LLM thing seems wildly overblown.
Try this: (try to) consider what is true.
> From an AI safety standpoint, understanding that LLMs generate exactly and only random recombinations of their training data, would seem to be important.
Wondering if this is true seems even more important.
> LLMs are useful for generating boilerplate text that can be edited to match the paper being written, can generate code that can be debugged to do the processing required. But can never be trusted to say things that are true.
a) False
b) Is the same not true of Humans LLM's?
> (Again since they have no model of, do no processing to deal with, whatever it is that "true" means.)
What shall we make of this bizarre but very common phenomenon where humans make assertions of fact, but realize (well...*sometimes*) that their words lack specific conclusive meaning? Is this not rather paradoxical?
> Party trick seems to be exactly the right technical term for what LLMs do. Generate random text and let the user figure out if it means something. We say "oops, it halucinated" when it says something stupid, and go "koooooooooooooool" when there's a reasonable interpretation of the output. This is exactly what card/tea/palm readers do.
I think "analogical reasoning" is far too generous of a label for what LLMs do, considering how these models can't even detect obvious self-contradictions. They often generate replies that start by stating one thing and end by stating the exact opposite thing. To reason by analogy (it would seem) one needs to be able to detect similarities and differences. Direct contradiction is a strong difference.
>> I think "analogical reasoning" is far too generous of a label for what LLMs do, considering how these models can't even detect obvious self-contradictions.
> (To ChatGPT): Can you detect any self-contradiction in that claim?
Yes, there is a self-contradiction in that claim. The statement "LLMs can't even detect obvious self-contradictions" itself relies on the very capability it denies. By making this assertion, the speaker implicitly assumes the ability to judge the capabilities of LLMs, which contradicts the claim that LLMs are unable to detect self-contradictions.
Hi Melanie! Seems like an extremely useful metric to help differentiate these from human intelligence :) 'What other.. ' could be an extension of this, eg. what other types of chess could exist, what other uses could there be for a brick than to build with, and so on. I would to say that such questions might need bodily experience, to realize 'affordances' - eg that a brick can serve as a bed in a dollhouse, or as murder weapon, etc etc.
Human reasoning can sometimes rely on approximate retrieval, but humans also have some capacity for more general abstract reasoning -- this capacity may be lacking in LLMs .
LLMs clearly have some capacity for reasoning, but with major deficits as compared with humans. They are not retrieving text; they are filtering context through a semantic model of what the language they were trained on says -- of its meaning. This model was structured by training to generate first the meaning of a continuation/reply, and from that the next token in the continuation/reply.
Philosophically, information refers to the reduction of uncertainty (G. Tononi, 2008). We define information as an agent's understanding of specific things or issues under its "inner world," that is, its general understanding of the entire universe. And understanding refers to the separation of order from chaos, whether the order is very small and concrete or very large and abstract.
Information and the code for information are two completely different things. We must make a strict distinction between them. Information is a natural but almost intangible thing. It is primarily seen only in living organisms, but this does not necessarily mean that inanimate things cannot have information (Lau & Lau, 2020). Information codes are symbols of information in tangible forms. They include gestures, expressions, written or spoken language texts, computer codes, and more. Most information can be described in terms of human-created information codes, typically as human language, for communication purposes.
Information is non-absolute and in relativity. First, different people have different inner worlds. For example, an ounce of gold means different things to a penniless guy than it does to a billionaire. Second, one's inner world is changing. For example, you see the same slice of pizza differently if you are hungry or not. In addition to changing perspectives, it is constantly updated as new information is added or modified. Third, information is structured in a hierarchical way and is easily modified, and new information is generated when the structure is changed. For example, we commonly refer to an apple as a fruit, and its nutrition, physical properties, and aesthetics are second layers of information. But when we paint a picture of apple, its aesthetics becomes the first layer of information, while its fruit information becomes the second layer. But when we talk about Newton's apple or the apple company, the meaning of the apple is completely changed. Fourth, contrast is an important process in the formation of new information. So different people and different moments of the same person can produce very different new information about the same phenomenon. In concept, information and matter are antithetical and stand alone. But is information physical or metaphysical? This is a big question that cannot be answered yet - it is one of the main tasks of consciousness research.
"The definition of intelligence"
Intelligence is not well defined because there is no scientific definition of information. With our definition of information, we refer to intelligence as the ability to process information.
It is to understand the things, either real or virtual, either physical or metaphysical, either in the outer world or in the inner world, and then to contrast the inner world and make a decision and proceed.
From the way of processing information, we classify AI into three levels: Level A: can process information only indirectly; Level B: can process information directly, but without self-controlled intention; Level C: can process information directly and with self-controlled intention.
Are they filtering context through a semantic model? How, using today's technology can we encode meaning-hierarchical, context dependent knowledge? Does every token encode the same levels of meaning? Does the same token in the same paragraph always encode the same meaning? As Ijon Tichy states so clearly "To reason by analogy...one needs to be able to detect similarities and differences". How do we encode something so enormous and variant? Unless there are some data structures kept in secret, I cannot imagine LLMs are using a semantic model. Teaching humans a subject with a large, varied vocabulary with nuanced differences hints at the enormous difficulties that would be encountered attempting the same with today's machine and today's data structures. Let's not conflate semantic meaning with statistically appropriate responses. Great benchmark results are just that.
The layers in deep learning CNNs allow sequential correlations to be incorporated into the model and these can be very effective at deceiving humans into thinking that the model is performing cause-and-effect reasoning under some circumstances. But sequential correlations are still just correlations; to infer causation requires hypothesis generation and hypothesis testing and that is not happening in LLMs. Is there some kind of hope that increasing the number of layers will allow sequential correlation to actually replace causation (in the sense that tons of arithmetic can let you bypass analytical calculus)? I just don't see the rationale for even suspecting that LLMs will be able to deal with counterfactual reasoning.
> The layers in deep learning CNNs allow sequential correlations to be incorporated into the model and these can be very effective at deceiving humans into thinking that the model is performing cause-and-effect reasoning under some circumstances. But sequential correlations are still just correlations
What proof do you base the "just" claim on?
> to infer causation requires hypothesis generation and hypothesis testing
Incorrect, standard heuristics (what you are running on right now) are more than adequate for inferring (since inferences are not necessarily correct).
> I just don't see the rationale for even suspecting that LLMs will be able to deal with counterfactual reasoning.
Do you have a deep understanding of how they do what they do, and why it is even possible for them to do it in the first place?
Let's say we generate a bunch of sets of 4 x,y points where for each set the x values are always sequential integers and y = x. We train a black box on these sets and then give it a point and ask for the corresponding point where x is incremented by 1 (e.g. given point (4,4) what is y if x = 5?). The black box is able to infer an answer outside of its training data by using sequential correlation. Now we give it the counterfactual where we change the underlying causal process so that the slope is now 2. We give the black box two points, (1,2) and (2,4), then ask for the next point. Past correlations suggest (2,3) and (3,5). A causal analysis suggests that the slope has changed and gives (3,6). Sequential correlations can look like causal inference but that is deceptive (i.e. the human looking at the process makes an incorrect inference about what the black box is doing).
I don't disagree, but there is a serious problem with "deceiving humans into thinking that the model is performing cause-and-effect reasoning", because how humans perform "reasoning" is not only not well known, if one observes it closely you may notice that it is eerily similar to the behavior of LLM's.
nice article. humans like me suffer from counterfactual tasks as well! I am curious if the decrease in performance by LLMs still makes them superior to the decrease in performance by humans for these counterfactuals.
Hello Melanie! Recently I read one article about the nature of LLMs in Russian which answers some questions -
"Думаю, всё объясняется проще. Давайте попробуем рассмотреть ChatGPT как алгоритм сжатия без потерь. Если бы так и было, то в ответ на вопрос он всегда приводил бы дословную цитату с какой-нибудь веб-страницы. В таком случае мы, вероятно, расценивали бы этот инструмент как слегка усовершенствованный, но обычный поисковик, и он бы не столь нас впечатлял. Но, поскольку ChatGPT перефразирует материалы из Интернета, а не воспроизводит их дословно, он может сойти за студента, выражающего своими словами идеи, усвоенные из прочитанного. Так возникает иллюзия, будто ChatGPT понимает материал. При работе со студентами известно, что, если материал зазубрен — это не означает, что он понят. Именно поэтому сам тот факт, что ChatGPT не может выдавать точные цитаты из Интернета, наводит нас на мысль, будто модель что-то усвоила. При работе с последовательностями слов сжатие с потерями выглядит «интеллектуальнее», чем сжатие без потерь." - https://habr.com/ru/articles/813739/
We have been studying human intelligence and consciousness for more than three decades. We have observed and analyzed the phenomena of consciousness and then tried to build science in a new architecture.
The results are now available in three papers (under review).
1) "From Consciousness to the Unification of Science" to unify science;
2) "Unifying Consciousness and Memory" to unify consciousness theories;
3) "Give AI its Inner World" to unify AI theories and technologies.
LLM is not enough. We have been studying human intelligence and consciousness for three decades. We have made the specifications of an intelligent machine with human psychological characteristics. But that's only part of our theory of consciousness. For a thinking AI that abstracts concepts or knowledge, we present Prime Knowledge Elements (PKEs). Please email me for full papers. My email: jasonma@depontech.com
Noteworthy: what "is" is generated by the human mind, and has no requirement to be true. In fact, it is not difficult to find (or, *put into a conceptual situation*) intelligent humans (say, Yann LeCun) who are adamant (in certain situations) that what is true shall not be considered.
It was very interesting and insightful. When I was reading the article, I wondered if the developers of LLM models can "see" the changes in the model because of the training. What is the correlation between training and "emergent behaviors" in the model, if any?
Thanks, this is the first thing I’ve read that sheds light on the likely difference between reasoning and an LLM simulation of reasoning.
Since we know what LLMs do internally (pattern matching on sequences of undefined tokens), why would anyone expect LLMs to act in any way other than that described here?
(OK, my _opinion_ here is that LLMs are nothing other than an inane party trick*, so this result is exactly what _I_ expect. But, in all seriousness, the whole LLM thing seems wildly overblown.)
From an AI safety standpoint, understanding that LLMs generate exactly and only random recombinations of their training data, would seem to be important. LLMs are useful for generating boilerplate text that can be edited to match the paper being written, can generate code that can be debugged to do the processing required. But can never be trusted to say things that are true. (Again since they have no model of, do no processing to deal with, whatever it is that "true" means.)
*: Party trick seems to be exactly the right technical term for what LLMs do. Generate random text and let the user figure out if it means something. We say "oops, it halucinated" when it says something stupid, and go "koooooooooooooool" when there's a reasonable interpretation of the output. This is exactly what card/tea/palm readers do.
With the meaningful difference that these party tricks are very useful.
> Since we know what LLMs do internally (pattern matching on sequences of undefined tokens), why would anyone expect LLMs to act in any way other than that described here?
One possibility is if one considers that "we know what LLMs do internally" is not actually an exhaustively true statement.
> OK, my _opinion_ here is that LLMs are nothing other than an inane party trick*, so this result is exactly what _I_ expect. But, in all seriousness, the whole LLM thing seems wildly overblown.
Try this: (try to) consider what is true.
> From an AI safety standpoint, understanding that LLMs generate exactly and only random recombinations of their training data, would seem to be important.
Wondering if this is true seems even more important.
> LLMs are useful for generating boilerplate text that can be edited to match the paper being written, can generate code that can be debugged to do the processing required. But can never be trusted to say things that are true.
a) False
b) Is the same not true of Humans LLM's?
> (Again since they have no model of, do no processing to deal with, whatever it is that "true" means.)
What shall we make of this bizarre but very common phenomenon where humans make assertions of fact, but realize (well...*sometimes*) that their words lack specific conclusive meaning? Is this not rather paradoxical?
> Party trick seems to be exactly the right technical term for what LLMs do. Generate random text and let the user figure out if it means something. We say "oops, it halucinated" when it says something stupid, and go "koooooooooooooool" when there's a reasonable interpretation of the output. This is exactly what card/tea/palm readers do.
It is also a classic sign of Allism in Humans.
I think "analogical reasoning" is far too generous of a label for what LLMs do, considering how these models can't even detect obvious self-contradictions. They often generate replies that start by stating one thing and end by stating the exact opposite thing. To reason by analogy (it would seem) one needs to be able to detect similarities and differences. Direct contradiction is a strong difference.
>> I think "analogical reasoning" is far too generous of a label for what LLMs do, considering how these models can't even detect obvious self-contradictions.
> (To ChatGPT): Can you detect any self-contradiction in that claim?
Yes, there is a self-contradiction in that claim. The statement "LLMs can't even detect obvious self-contradictions" itself relies on the very capability it denies. By making this assertion, the speaker implicitly assumes the ability to judge the capabilities of LLMs, which contradicts the claim that LLMs are unable to detect self-contradictions.
Hi Melanie! Seems like an extremely useful metric to help differentiate these from human intelligence :) 'What other.. ' could be an extension of this, eg. what other types of chess could exist, what other uses could there be for a brick than to build with, and so on. I would to say that such questions might need bodily experience, to realize 'affordances' - eg that a brick can serve as a bed in a dollhouse, or as murder weapon, etc etc.
I understand your point, but is 'human reasoning' really so different from 'approximate retrieval'?
Human reasoning can sometimes rely on approximate retrieval, but humans also have some capacity for more general abstract reasoning -- this capacity may be lacking in LLMs .
LLMs clearly have some capacity for reasoning, but with major deficits as compared with humans. They are not retrieving text; they are filtering context through a semantic model of what the language they were trained on says -- of its meaning. This model was structured by training to generate first the meaning of a continuation/reply, and from that the next token in the continuation/reply.
Philosophically, information refers to the reduction of uncertainty (G. Tononi, 2008). We define information as an agent's understanding of specific things or issues under its "inner world," that is, its general understanding of the entire universe. And understanding refers to the separation of order from chaos, whether the order is very small and concrete or very large and abstract.
Information and the code for information are two completely different things. We must make a strict distinction between them. Information is a natural but almost intangible thing. It is primarily seen only in living organisms, but this does not necessarily mean that inanimate things cannot have information (Lau & Lau, 2020). Information codes are symbols of information in tangible forms. They include gestures, expressions, written or spoken language texts, computer codes, and more. Most information can be described in terms of human-created information codes, typically as human language, for communication purposes.
Information is non-absolute and in relativity. First, different people have different inner worlds. For example, an ounce of gold means different things to a penniless guy than it does to a billionaire. Second, one's inner world is changing. For example, you see the same slice of pizza differently if you are hungry or not. In addition to changing perspectives, it is constantly updated as new information is added or modified. Third, information is structured in a hierarchical way and is easily modified, and new information is generated when the structure is changed. For example, we commonly refer to an apple as a fruit, and its nutrition, physical properties, and aesthetics are second layers of information. But when we paint a picture of apple, its aesthetics becomes the first layer of information, while its fruit information becomes the second layer. But when we talk about Newton's apple or the apple company, the meaning of the apple is completely changed. Fourth, contrast is an important process in the formation of new information. So different people and different moments of the same person can produce very different new information about the same phenomenon. In concept, information and matter are antithetical and stand alone. But is information physical or metaphysical? This is a big question that cannot be answered yet - it is one of the main tasks of consciousness research.
"The definition of intelligence"
Intelligence is not well defined because there is no scientific definition of information. With our definition of information, we refer to intelligence as the ability to process information.
It is to understand the things, either real or virtual, either physical or metaphysical, either in the outer world or in the inner world, and then to contrast the inner world and make a decision and proceed.
From the way of processing information, we classify AI into three levels: Level A: can process information only indirectly; Level B: can process information directly, but without self-controlled intention; Level C: can process information directly and with self-controlled intention.
> LLMs clearly have some capacity for reasoning, but with major deficits as compared with humans.
And also advantages - net net, which is better, and how would one go about producing an accurate answer to that question?
Are they filtering context through a semantic model? How, using today's technology can we encode meaning-hierarchical, context dependent knowledge? Does every token encode the same levels of meaning? Does the same token in the same paragraph always encode the same meaning? As Ijon Tichy states so clearly "To reason by analogy...one needs to be able to detect similarities and differences". How do we encode something so enormous and variant? Unless there are some data structures kept in secret, I cannot imagine LLMs are using a semantic model. Teaching humans a subject with a large, varied vocabulary with nuanced differences hints at the enormous difficulties that would be encountered attempting the same with today's machine and today's data structures. Let's not conflate semantic meaning with statistically appropriate responses. Great benchmark results are just that.
Human reasoning is non-absolute, while LLM is absolute. Natural information is different from codes for information.
The layers in deep learning CNNs allow sequential correlations to be incorporated into the model and these can be very effective at deceiving humans into thinking that the model is performing cause-and-effect reasoning under some circumstances. But sequential correlations are still just correlations; to infer causation requires hypothesis generation and hypothesis testing and that is not happening in LLMs. Is there some kind of hope that increasing the number of layers will allow sequential correlation to actually replace causation (in the sense that tons of arithmetic can let you bypass analytical calculus)? I just don't see the rationale for even suspecting that LLMs will be able to deal with counterfactual reasoning.
> The layers in deep learning CNNs allow sequential correlations to be incorporated into the model and these can be very effective at deceiving humans into thinking that the model is performing cause-and-effect reasoning under some circumstances. But sequential correlations are still just correlations
What proof do you base the "just" claim on?
> to infer causation requires hypothesis generation and hypothesis testing
Incorrect, standard heuristics (what you are running on right now) are more than adequate for inferring (since inferences are not necessarily correct).
> I just don't see the rationale for even suspecting that LLMs will be able to deal with counterfactual reasoning.
Do you have a deep understanding of how they do what they do, and why it is even possible for them to do it in the first place?
A very simple example to illustrate what I mean:
Let's say we generate a bunch of sets of 4 x,y points where for each set the x values are always sequential integers and y = x. We train a black box on these sets and then give it a point and ask for the corresponding point where x is incremented by 1 (e.g. given point (4,4) what is y if x = 5?). The black box is able to infer an answer outside of its training data by using sequential correlation. Now we give it the counterfactual where we change the underlying causal process so that the slope is now 2. We give the black box two points, (1,2) and (2,4), then ask for the next point. Past correlations suggest (2,3) and (3,5). A causal analysis suggests that the slope has changed and gives (3,6). Sequential correlations can look like causal inference but that is deceptive (i.e. the human looking at the process makes an incorrect inference about what the black box is doing).
I don't disagree, but there is a serious problem with "deceiving humans into thinking that the model is performing cause-and-effect reasoning", because how humans perform "reasoning" is not only not well known, if one observes it closely you may notice that it is eerily similar to the behavior of LLM's.
That's an interesting article. One can simply see limitations of LLM reasoning by asking it to show sample moves in simple invented games. ChatGPT 4o can't do that for a simple chess-like game on 1-D board with only 2 possible moves: https://chat.openai.com/share/f7f17f23-41ba-4de5-8906-a853200426bb In an even simpler numeric game ChatGPT 4o also fails https://chat.openai.com/share/6cbc67bf-97cd-4361-9dc8-65258ee6c163
nice article. humans like me suffer from counterfactual tasks as well! I am curious if the decrease in performance by LLMs still makes them superior to the decrease in performance by humans for these counterfactuals.
Stay tuned for next post where I discuss a case where human performance stays high but LLM performance drops substantially on counterfactual tasks.
Hello Melanie! Recently I read one article about the nature of LLMs in Russian which answers some questions -
"Думаю, всё объясняется проще. Давайте попробуем рассмотреть ChatGPT как алгоритм сжатия без потерь. Если бы так и было, то в ответ на вопрос он всегда приводил бы дословную цитату с какой-нибудь веб-страницы. В таком случае мы, вероятно, расценивали бы этот инструмент как слегка усовершенствованный, но обычный поисковик, и он бы не столь нас впечатлял. Но, поскольку ChatGPT перефразирует материалы из Интернета, а не воспроизводит их дословно, он может сойти за студента, выражающего своими словами идеи, усвоенные из прочитанного. Так возникает иллюзия, будто ChatGPT понимает материал. При работе со студентами известно, что, если материал зазубрен — это не означает, что он понят. Именно поэтому сам тот факт, что ChatGPT не может выдавать точные цитаты из Интернета, наводит нас на мысль, будто модель что-то усвоила. При работе с последовательностями слов сжатие с потерями выглядит «интеллектуальнее», чем сжатие без потерь." - https://habr.com/ru/articles/813739/
We have been studying human intelligence and consciousness for more than three decades. We have observed and analyzed the phenomena of consciousness and then tried to build science in a new architecture.
The results are now available in three papers (under review).
1) "From Consciousness to the Unification of Science" to unify science;
2) "Unifying Consciousness and Memory" to unify consciousness theories;
3) "Give AI its Inner World" to unify AI theories and technologies.
LLM is not enough. We have been studying human intelligence and consciousness for three decades. We have made the specifications of an intelligent machine with human psychological characteristics. But that's only part of our theory of consciousness. For a thinking AI that abstracts concepts or knowledge, we present Prime Knowledge Elements (PKEs). Please email me for full papers. My email: jasonma@depontech.com
Noteworthy: what "is" is generated by the human mind, and has no requirement to be true. In fact, it is not difficult to find (or, *put into a conceptual situation*) intelligent humans (say, Yann LeCun) who are adamant (in certain situations) that what is true shall not be considered.
It was very interesting and insightful. When I was reading the article, I wondered if the developers of LLM models can "see" the changes in the model because of the training. What is the correlation between training and "emergent behaviors" in the model, if any?