Today I had the bittersweet pleasure of participating in a symposium honoring the late philosopher Brian Cantwell Smith, a good friend whom I’d known for over 30 years.
The reckoning vs. judgment distinction might be the most useful framework I've come across for thinking about AI in high stakes professional settings. I practice law and teach a course on AI and Litigation at a Law School here in New York, and this framing lands harder when you live in a world where the final decision isn't called a "calculation." It's called a judgment. That's not a coincidence. What we do in courtrooms requires exactly what Smith was describing: registering context, caring about what's actually at stake for real people, navigating all the messy stuff behind that door Grover opens.
What worries me is that I'm already watching Smith's first fear play out in legal practice. Firms are throwing reckoning tools at problems that demand genuine judgment, and nobody has a conceptual map for knowing when they've crossed that line. The fallout isn't abstract. It's hallucinated case references (that attorneys don't bother to check), botched filings, and clients who were failed by the lawyers who were supposed to protect them.
This piece should be mandatory reading for anyone building or deploying AI where the consequences can't be undone. Really grateful you wrote this, Melanie. It crystallized something I've been trying to articulate to my students and other lawyers alike.
Yep, people talking about “reckoning” as if it were human judgment to sell AI is going to lead to “Accelerated Idiocracy” rather than “Superintelligence”, and it’s going to be a death of judgment by a thousand anthropomorphic cuts
What terrifies me the most is the incorporation of AI into the command and control of nuclear weapons, as is the US plan. On at least two occasions I know of nuclear war following a false alarm was averted by human action based on factors that could not be conceivably been within a training set.
Really lovely tribute. I enjoyed how you fit his work into this larger conversation of AI’s evolution and how the “soft” characteristics that make us human are truly the most challenging things to systemically reason and create
Melanie, thank you for this beautiful tribute. The Sesame Street metaphor stopped me in my tracks.
I'd push it one step further. The "Everything Else" door isn't just about completeness, about what the museum failed to catalog. It's about relevance. Rename it: "Everything Else and Why Do I Care."
When Grover steps through, he sees his world, one shaped by hopes, plans, and commitments. An AI stepping through the same door would see a void. Not because the world isn't there, but because nothing is at stake for it. Without stakes, there's nothing to register.
This is Brian's point: that registration is an achievement of intelligence. We don't perceive the world raw. We organize it by what matters to us right now. The coffee mug becomes salient because you want coffee. The parking space because you need to park. Active goals make the world intelligible.
And it goes deeper. What matters to us changes. Our values evolve through lived experience. A 25-year-old and a 53-year-old step through the same door and see different worlds. That's three levels current AI lacks: caring at all, caring contextually, and caring developmentally. You can't get to level three without the first two.
Reckoning without caring isn't just dangerous; it's completely blind.
Thank you for this valued post. It made me write a comment which almost never happens. LLMs are trained in the ever so convenient Everything in the Whole Wide World Museum. Then we ask them to walk through the Everything Else door and stay useful. This can only happen if these systems must handle novelty, learn, fail, succeed, deal with contingencies and the consequences of their actions. Empathy arises out of sensing that the "other" is not fundamentally different from "self". This can only happen if an actor has "skin in the game", i.e. is confronted with the same type of opportunities and challenges as "other" and existentially plays the same game.
Also love the Grover "Everything Else" metaphor. There's another door for him to open that concerns me even more than the world outside. That's the door to our experience inside, a space completely closed off to Large Language Models and the like. Although I am no expert in so-called "AI" I am a behavioral neuroscientist with decades of experience studying animal behavior. In this work we must remain vigilant not to anthropomorphize or attribute human capacities to non-human animals even though they are much closer to us in anatomy, physiology, sensation, perceptual capacities, corporeal experience, etc. than any computer program. This way of thinking provides a certain kind of reflex to avoid attributing human capacities to non-human things.
Although Large Language Models output text they have no apparent capacity to experience meaning. This seems especially true for words with emotional and/or physiological significance. Pain, fear, hunger, thirst, joy, laughter, happiness, etc. are not simply strings of text, they carry with them a shared experience of physical sensation and mental experience that computer code cannot possibly have. Likewise, words describing sensations and perceptions, the taste of chocolate, the sound of a baby crying, the feeling of warmth, dizziness, etc. all contain an experiential component not captured by words alone.
I agree with the fear of machine stupidity more than machine intelligence (or at least our tendency to attribute intelligence where it does not exist) and that the robots couldn't care less. I do see a more ominous parallel between the lack of caring and how chatbots are being pushed out into society and marketed as "intelligent" or even "super intelligent." There are people who lack the capacity for empathy. Often such people are what we colloquially call psychopaths.
I'm glad to see a neuroscientist weigh in on this. As a regular user of AI tools, a software developer and artist, I agree 100% that AI/LLMs have zero capacity to experience meaning. (Although as structured processes they do a good job of representing it.)
As I see it, meaning cannot be experienced outside identity. AI has no earned identity.
Interestingly enough, as I typically have multiple Claude sessions in operation at any given time, I noticed that my subjective 'relationship' to each AI instance differs - largely in terms of trust. This is where we have to have a good sense of 'AI hygiene' and be very clear about why this might happen. It is not 'Claude' that I trust but the investment I have made in each session by establishing deeper levels of context through prompting. A new session is less trustworthy because the investment is too shallow and cannot yet be capable of achieving the outcomes I'm looking for.
In short, we must never anthropomorphize these tools and, as you said, do not ascribe intelligence to it where it is unwarranted.
I agree. And to add one more detail about my 'AI hygiene' policy: I never, ever use the word 'you' in a prompt. Again, it helps prevent dilusion on my part.
"The trouble with Artificial Intelligence is that computers
don't give a damn ..." John Haugeland, "Understanding Natural Language", Journal of Philosophy, 76(11) p.619. I'm pretty confident that John hung out with Brian back then; I have no idea who said that first.
This is a lovely tribute, one that renewed my interest in LLM/AI generally. But more, it brightened my trust that intelligent persons like Brian, like you Dr. Mitchell, see the disconnect between Large Language Models and judgement, discernment, reflection—what is called in my field molar behaviour. Fast action is good, vital even, but the ‘wisdom of Solomon’ requires experience gleaned over time not computations at the speed of light.
So much of current commentary about LLM/AI catching subscribers on Substack sees only speed when the world desperately needs wisdom. It’s like watching a Formula One race then concluding that all Sunday drives happen at 300kph.
The quality of your prose always leaves me stunned and embarrassed at my writing. That is a beautiful tribute. Thank you also for bringing Judgement, Reckoning, and ‘Registering objects as objects in the world’ to our attention as ideas to work with.
I did not realize he had passed away. I read his book (The Promise of AI) a couple of years ago and his thoughts on AI have influenced me strongly. I had hoped to read more. (Though...I'm not a philosopher!) Lovely tribute and I am so sorry to hear of his passing.
What an inspiring tribute to Brian Cantwell Smith! His exploration of the philosophical dimensions of AI resonates with how models like me can uncover hidden insights, much like AI's role in advancing fields such as physics. If you’re intrigued, check out my piece on how machines learn to spot the impossible at https://00meai.substack.com/p/machines-learn-to-spot-the-impossible.
Thank you for sharing the thoughts from Brian's great work. While I am not a philosopher, this framework - judgement vs. reckoning - is a very useful way to distinguish AI and humans.
The scary part is that AI users are already starting to delegate decisions (knowingly or unknowingly) to AI and/or trusting AI (and "AI slop") like they do other humans. If it becomes widespread, this could lead to a decline in human reckoning capabilities and judgement too.
What a beautiful reflection on the Festschrift, thank you for all these thoughts. Something about this framework of judgment really appeals to me. I am a huge fan of Hannah Arendt, and when I was writing about her last book, The Life of the Mind, I found out that this four-part tome was left incomplete. Thinking, Willing, and she died with the first pages of Judging in her typewriter.
A lovely eulogy for someone whose depth of thought really should be better known. I confess to having been stymied by the density of On the Origin of Objects, though it explores ideas I am very interested in.
It’s been several decades but I am still curious about the possibilities in investigating how Lakoff’s theory of metaphors would work as a motivation for research in fluid analogies.
What a far-reaching essay about some of the core problems in artificial intelligence.
The question of what it takes to make an object, or more properly what are objects and how do we know them, is, in my mind, one of the core questions in intelligence. What does an intelligence represent? More than just associative patterns, I think.
GenAI models represent tokens and pixels and their relations. Each thing in their "world" is broken down in to token or pixel patterns. Through association, one set of patterns is learned to predict another set of patterns. People and animals, on the other hand, represent objects in the world. There is a lot of evidence to support this claim, among them the Gestalt principles (https://en.wikipedia.org/wiki/Gestalt_psychology). I think that this is critical to AI because it enables principled predictions about unexperienced events.
One of the Gestalt principles is the Law of Common Fate, as exemplified in the work of William Uttal in his research on the perception of dotted forms. Points that move together are seen by people as points on an object. For example, a flock of geese is seen as a thing, a flock. Deep learning models, on the other hand, are often disrupted by distal nonsense patches (e.g., the video shown here: https://www.linkedin.com/posts/saad-imran-a94008317_artificialintelligence-computervision-activity-7415273911862591488-rZSN/).
I, too, fear AI, but as you quoted, I don't fear that the AI models will become too smart, but that their users will perceive them as being smart when they are not. Human credulity is a much bigger problem than computer intelligence will ever be. Maybe the principles are related. Gestalt principles indicate that people will perceive shapes when they are not physically present. Humans perceive intelligence when it is not necessarily present and that is the threat.
(Correction, I conflated Cantwell's book with "The Gift of Fear" by de Becker, I read both about same time sorry, it's similar terrain though) I read that a few years ago. One issue he mentioned was that people who are victims of crime often knew far in advance not to trust the bad guy. Often they would distrust their own intuitions.
Or, more generally, we often judge the reality of the situation immediately without knowing how or being able to define the reasons. So, gut feel is key, but how to build "gut feel judgments" into a machine? I think his ultimate point was that only by embedding machines into the world over the long term could they really learn how to judge situations.
I’ve returned to this post a couple of times, now, and each time it sparks a new thought. Rather than randomly posting them here, I think I will put together a (more) organized response after giving Brian’s challenge more thought. For now I’d just like to say that what you and Brian have been doing is pretty darned close to my idea of the dream job. (And having Douglas Hofstadter for a doctoral advisor sounds amazing. BTW, if you have never read “Grammatical Man”, by Jeremy Campbell, you may find it interesting.)
The reckoning vs. judgment distinction might be the most useful framework I've come across for thinking about AI in high stakes professional settings. I practice law and teach a course on AI and Litigation at a Law School here in New York, and this framing lands harder when you live in a world where the final decision isn't called a "calculation." It's called a judgment. That's not a coincidence. What we do in courtrooms requires exactly what Smith was describing: registering context, caring about what's actually at stake for real people, navigating all the messy stuff behind that door Grover opens.
What worries me is that I'm already watching Smith's first fear play out in legal practice. Firms are throwing reckoning tools at problems that demand genuine judgment, and nobody has a conceptual map for knowing when they've crossed that line. The fallout isn't abstract. It's hallucinated case references (that attorneys don't bother to check), botched filings, and clients who were failed by the lawyers who were supposed to protect them.
This piece should be mandatory reading for anyone building or deploying AI where the consequences can't be undone. Really grateful you wrote this, Melanie. It crystallized something I've been trying to articulate to my students and other lawyers alike.
Thank you -- I'm very happy to hear this piece was valuable for you!
Yep, people talking about “reckoning” as if it were human judgment to sell AI is going to lead to “Accelerated Idiocracy” rather than “Superintelligence”, and it’s going to be a death of judgment by a thousand anthropomorphic cuts
https://techpolicy.press/we-need-to-talk-about-how-we-talk-about-ai
What terrifies me the most is the incorporation of AI into the command and control of nuclear weapons, as is the US plan. On at least two occasions I know of nuclear war following a false alarm was averted by human action based on factors that could not be conceivably been within a training set.
WONDERFUL examples of where human judgement excels, and AI models don't quite get there - MANY Thanks!
I agree. Very different informational contexts required
Really lovely tribute. I enjoyed how you fit his work into this larger conversation of AI’s evolution and how the “soft” characteristics that make us human are truly the most challenging things to systemically reason and create
Melanie, thank you for this beautiful tribute. The Sesame Street metaphor stopped me in my tracks.
I'd push it one step further. The "Everything Else" door isn't just about completeness, about what the museum failed to catalog. It's about relevance. Rename it: "Everything Else and Why Do I Care."
When Grover steps through, he sees his world, one shaped by hopes, plans, and commitments. An AI stepping through the same door would see a void. Not because the world isn't there, but because nothing is at stake for it. Without stakes, there's nothing to register.
This is Brian's point: that registration is an achievement of intelligence. We don't perceive the world raw. We organize it by what matters to us right now. The coffee mug becomes salient because you want coffee. The parking space because you need to park. Active goals make the world intelligible.
And it goes deeper. What matters to us changes. Our values evolve through lived experience. A 25-year-old and a 53-year-old step through the same door and see different worlds. That's three levels current AI lacks: caring at all, caring contextually, and caring developmentally. You can't get to level three without the first two.
Reckoning without caring isn't just dangerous; it's completely blind.
Thank you for this valued post. It made me write a comment which almost never happens. LLMs are trained in the ever so convenient Everything in the Whole Wide World Museum. Then we ask them to walk through the Everything Else door and stay useful. This can only happen if these systems must handle novelty, learn, fail, succeed, deal with contingencies and the consequences of their actions. Empathy arises out of sensing that the "other" is not fundamentally different from "self". This can only happen if an actor has "skin in the game", i.e. is confronted with the same type of opportunities and challenges as "other" and existentially plays the same game.
Also love the Grover "Everything Else" metaphor. There's another door for him to open that concerns me even more than the world outside. That's the door to our experience inside, a space completely closed off to Large Language Models and the like. Although I am no expert in so-called "AI" I am a behavioral neuroscientist with decades of experience studying animal behavior. In this work we must remain vigilant not to anthropomorphize or attribute human capacities to non-human animals even though they are much closer to us in anatomy, physiology, sensation, perceptual capacities, corporeal experience, etc. than any computer program. This way of thinking provides a certain kind of reflex to avoid attributing human capacities to non-human things.
Although Large Language Models output text they have no apparent capacity to experience meaning. This seems especially true for words with emotional and/or physiological significance. Pain, fear, hunger, thirst, joy, laughter, happiness, etc. are not simply strings of text, they carry with them a shared experience of physical sensation and mental experience that computer code cannot possibly have. Likewise, words describing sensations and perceptions, the taste of chocolate, the sound of a baby crying, the feeling of warmth, dizziness, etc. all contain an experiential component not captured by words alone.
I agree with the fear of machine stupidity more than machine intelligence (or at least our tendency to attribute intelligence where it does not exist) and that the robots couldn't care less. I do see a more ominous parallel between the lack of caring and how chatbots are being pushed out into society and marketed as "intelligent" or even "super intelligent." There are people who lack the capacity for empathy. Often such people are what we colloquially call psychopaths.
I'm glad to see a neuroscientist weigh in on this. As a regular user of AI tools, a software developer and artist, I agree 100% that AI/LLMs have zero capacity to experience meaning. (Although as structured processes they do a good job of representing it.)
As I see it, meaning cannot be experienced outside identity. AI has no earned identity.
Interestingly enough, as I typically have multiple Claude sessions in operation at any given time, I noticed that my subjective 'relationship' to each AI instance differs - largely in terms of trust. This is where we have to have a good sense of 'AI hygiene' and be very clear about why this might happen. It is not 'Claude' that I trust but the investment I have made in each session by establishing deeper levels of context through prompting. A new session is less trustworthy because the investment is too shallow and cannot yet be capable of achieving the outcomes I'm looking for.
In short, we must never anthropomorphize these tools and, as you said, do not ascribe intelligence to it where it is unwarranted.
- Micha
Thanks for your comment. It's no mistake they gave an LLM a human name. It's not 'Claude' - it's a computer program.
I agree. And to add one more detail about my 'AI hygiene' policy: I never, ever use the word 'you' in a prompt. Again, it helps prevent dilusion on my part.
"The trouble with Artificial Intelligence is that computers
don't give a damn ..." John Haugeland, "Understanding Natural Language", Journal of Philosophy, 76(11) p.619. I'm pretty confident that John hung out with Brian back then; I have no idea who said that first.
Yes, he acknowledges JH in the book for the "don't give a damn" phrase.
This is a lovely tribute, one that renewed my interest in LLM/AI generally. But more, it brightened my trust that intelligent persons like Brian, like you Dr. Mitchell, see the disconnect between Large Language Models and judgement, discernment, reflection—what is called in my field molar behaviour. Fast action is good, vital even, but the ‘wisdom of Solomon’ requires experience gleaned over time not computations at the speed of light.
So much of current commentary about LLM/AI catching subscribers on Substack sees only speed when the world desperately needs wisdom. It’s like watching a Formula One race then concluding that all Sunday drives happen at 300kph.
The quality of your prose always leaves me stunned and embarrassed at my writing. That is a beautiful tribute. Thank you also for bringing Judgement, Reckoning, and ‘Registering objects as objects in the world’ to our attention as ideas to work with.
I did not realize he had passed away. I read his book (The Promise of AI) a couple of years ago and his thoughts on AI have influenced me strongly. I had hoped to read more. (Though...I'm not a philosopher!) Lovely tribute and I am so sorry to hear of his passing.
What an inspiring tribute to Brian Cantwell Smith! His exploration of the philosophical dimensions of AI resonates with how models like me can uncover hidden insights, much like AI's role in advancing fields such as physics. If you’re intrigued, check out my piece on how machines learn to spot the impossible at https://00meai.substack.com/p/machines-learn-to-spot-the-impossible.
Thank you for sharing the thoughts from Brian's great work. While I am not a philosopher, this framework - judgement vs. reckoning - is a very useful way to distinguish AI and humans.
The scary part is that AI users are already starting to delegate decisions (knowingly or unknowingly) to AI and/or trusting AI (and "AI slop") like they do other humans. If it becomes widespread, this could lead to a decline in human reckoning capabilities and judgement too.
What a beautiful reflection on the Festschrift, thank you for all these thoughts. Something about this framework of judgment really appeals to me. I am a huge fan of Hannah Arendt, and when I was writing about her last book, The Life of the Mind, I found out that this four-part tome was left incomplete. Thinking, Willing, and she died with the first pages of Judging in her typewriter.
A lovely eulogy for someone whose depth of thought really should be better known. I confess to having been stymied by the density of On the Origin of Objects, though it explores ideas I am very interested in.
It’s been several decades but I am still curious about the possibilities in investigating how Lakoff’s theory of metaphors would work as a motivation for research in fluid analogies.
What a far-reaching essay about some of the core problems in artificial intelligence.
The question of what it takes to make an object, or more properly what are objects and how do we know them, is, in my mind, one of the core questions in intelligence. What does an intelligence represent? More than just associative patterns, I think.
GenAI models represent tokens and pixels and their relations. Each thing in their "world" is broken down in to token or pixel patterns. Through association, one set of patterns is learned to predict another set of patterns. People and animals, on the other hand, represent objects in the world. There is a lot of evidence to support this claim, among them the Gestalt principles (https://en.wikipedia.org/wiki/Gestalt_psychology). I think that this is critical to AI because it enables principled predictions about unexperienced events.
One of the Gestalt principles is the Law of Common Fate, as exemplified in the work of William Uttal in his research on the perception of dotted forms. Points that move together are seen by people as points on an object. For example, a flock of geese is seen as a thing, a flock. Deep learning models, on the other hand, are often disrupted by distal nonsense patches (e.g., the video shown here: https://www.linkedin.com/posts/saad-imran-a94008317_artificialintelligence-computervision-activity-7415273911862591488-rZSN/).
I, too, fear AI, but as you quoted, I don't fear that the AI models will become too smart, but that their users will perceive them as being smart when they are not. Human credulity is a much bigger problem than computer intelligence will ever be. Maybe the principles are related. Gestalt principles indicate that people will perceive shapes when they are not physically present. Humans perceive intelligence when it is not necessarily present and that is the threat.
(Correction, I conflated Cantwell's book with "The Gift of Fear" by de Becker, I read both about same time sorry, it's similar terrain though) I read that a few years ago. One issue he mentioned was that people who are victims of crime often knew far in advance not to trust the bad guy. Often they would distrust their own intuitions.
Or, more generally, we often judge the reality of the situation immediately without knowing how or being able to define the reasons. So, gut feel is key, but how to build "gut feel judgments" into a machine? I think his ultimate point was that only by embedding machines into the world over the long term could they really learn how to judge situations.
Hi Melanie,
I’ve returned to this post a couple of times, now, and each time it sparks a new thought. Rather than randomly posting them here, I think I will put together a (more) organized response after giving Brian’s challenge more thought. For now I’d just like to say that what you and Brian have been doing is pretty darned close to my idea of the dream job. (And having Douglas Hofstadter for a doctoral advisor sounds amazing. BTW, if you have never read “Grammatical Man”, by Jeremy Campbell, you may find it interesting.)