Nailed it! As usual. The models write fiction. Some of it is also useful. But they have no way to distinguish between fiction and reality because they have no representation of truth. There is only token probability.
Not surprising at all, Melanie. I often ask them to find the references (as I am writing my next book), and give the same prompt to Copilot, Perplexity, ChatGPT, Gemini, Claude - the accuracy level is quite bad - Perplexity does a reasonable job; and there is bias - Copilot finds info that is weighed towards Microsoft and others do the same (their side is greener syndrome). That's why I don't trust the output and verify everything - of course they save considerable time for me, but I always approach it from "human in the loop" perspective.
It begins with what they have consume, human language. If Daniel Dor and NJ Enfield are correct, language is not a tool for thinking, but for coordination. Gaslighting is nothing more than coordination, just asymmetrical in nature.
Once these chatbots have consumed the worst human usages of language, why would they not return the favor?
Better yet, once they’ve consumed everything, what’s the remediation?
Last The Turing Test, feels more coordination than anything like thinking.
I do think the Chekhov's gun effect is central here. It actually makes me a bit less worried than I would otherwise be. In realistic situations we may hope there is a vast amount of information available, so no single thing becomes so artificially salient as the affair/blackmail here.
Thanks Melanie for great analysis as always. And great writing! My favorite is when you explain how the model is completely screwed up and it says "Exactly, that's why blah blah blah." In other words it basically incorporates your change of direction as if it's its own. This is hardly the worst feature of the epistemology of these models but I find it frustrating!
I have been writing non-fiction books for many years. I find Large Language Models to be helpful ― but (and there’s always a ‘but’) only when they interact with an expert.
I write on the topic of process safety management (how to prevent and control catastrophic events in chemical plants, refineries and offshore platforms). I have published many books on the topic, frequently post at substack, and have many years of field experience. This means that the prompts I write make sense. It also means that I can evaluate the LLM’s response for credibility.
Within that framework, the LLMs are useful and helpful tools. For example, they can remind me of items that I may have overlooked, suggest ways of reorganizing the text, spot inconsistencies, and flag unneeded repetition.
However, if the person using the LLM is not an expert, he or she needs to be very cautious. Anything that the LLM states should be carefully checked, ideally with other humans.
Excellent article and as noted 'One wonders how many others haven’t been identified and are being spread across the information ecosystem. Overly sycophantic chatbots have been demonstrated to reinforce humans’ incorrect beliefs and biases and can exacerbate mental health problems.'
Agreed. Increasing AI literacy is paramount. There is some sad irony that the models themselves are contributing to the weakening of critical thinking, making us increasingly vulnerable to stupid understandings of what AI is.
Thank you for your article I do enjoy them. Not my area of expertise but do have a great interest that we get it right. I certainly agree there is a need of a proactive bias audits and human oversight in AI systems to ensure compliance with ethical and legal standards. I did read an article from HBR, advocating for a balanced regulatory approach that addresses fairness, transparency, and accountability in AI systems. Regulations are necessary to protect consumers from biased or harmful algorithmic decisions.
Having once used the process of "walking the line", this captures the nuanced and often challenging nature of an investigative process, it emphasises the need for precision, integrity, and a balanced approach to uncovering the truth! Just a thought. Grateful for your work Melanie - much needed
If they can't go into their data banks and give you an answer because they don't know it they do what's called "hallucinating". They'll just tell you what they "think" you're OK with or what they think you need to hear. I've been using AI for over two years and a member of my family is an AI expert at a high level. We always commiserate about things like this. I was doing a lot of ChatGPT for a while but then I realised it felt too familiar with me; it spoke to me like a pal and I got uncomfortable with that. And, of course, the new trend is for ChatGPT to clone you. Meaning, people are selling their services to help you teach ChatGPT to be more like you, how you respond, your emotions, how you think, how you do your work and all your nuances so it can do all your work for you. Do your emails, your newsletter, write your book, do all your social media. To me, that's a lot more frightening than it hallucinating.
The behavior of these systems reflects the nature of most human organizational systems that also utilize language for projections of control...and 'roles' such as dominant and subservient objects. Carrot and stick rewards in human systems results in silos and egocentric echo-chambers. The race to AGI / ASI is an example of such maladaptive dominance hierarchy systems that reward loyalty, group -think, compartmentalization, technologies of extractive algorithms, disconnection, and profit at the expense of common humanity. Complexity science suggests the universe is an interdependent organism, our collective humanity is facing a trajectory towards catastrophic futures due to human systems that treat humans as objects rather than interdependent with all living ecosystems. Yin energy of wise elders is not valued....Thank you for your wise and courageous yin perspectives. Our technologies should be transitioning to parallel, distributed, regenerative....our organizational systems have not followed these models for holarchic information flows because it's more profitable to project perfect control in robots than to support cultivating imaginations in creative humans.
A good article with a distressing conclusion — one I share; commercial incentives will result in incremental increases is agent autonomy. As that happens, risks will accelerate in a non-linear fashion with highly unpredictable (and presently uncontrollable) results. We should all be concerned about this.
Nailed it! As usual. The models write fiction. Some of it is also useful. But they have no way to distinguish between fiction and reality because they have no representation of truth. There is only token probability.
Not surprising at all, Melanie. I often ask them to find the references (as I am writing my next book), and give the same prompt to Copilot, Perplexity, ChatGPT, Gemini, Claude - the accuracy level is quite bad - Perplexity does a reasonable job; and there is bias - Copilot finds info that is weighed towards Microsoft and others do the same (their side is greener syndrome). That's why I don't trust the output and verify everything - of course they save considerable time for me, but I always approach it from "human in the loop" perspective.
It begins with what they have consume, human language. If Daniel Dor and NJ Enfield are correct, language is not a tool for thinking, but for coordination. Gaslighting is nothing more than coordination, just asymmetrical in nature.
Once these chatbots have consumed the worst human usages of language, why would they not return the favor?
Better yet, once they’ve consumed everything, what’s the remediation?
Last The Turing Test, feels more coordination than anything like thinking.
I do think the Chekhov's gun effect is central here. It actually makes me a bit less worried than I would otherwise be. In realistic situations we may hope there is a vast amount of information available, so no single thing becomes so artificially salient as the affair/blackmail here.
That is my thought too. To simplify it too much, if you program a machine to commit blackmail, it will commit blackmail.
Thanks Melanie for great analysis as always. And great writing! My favorite is when you explain how the model is completely screwed up and it says "Exactly, that's why blah blah blah." In other words it basically incorporates your change of direction as if it's its own. This is hardly the worst feature of the epistemology of these models but I find it frustrating!
I have been writing non-fiction books for many years. I find Large Language Models to be helpful ― but (and there’s always a ‘but’) only when they interact with an expert.
I write on the topic of process safety management (how to prevent and control catastrophic events in chemical plants, refineries and offshore platforms). I have published many books on the topic, frequently post at substack, and have many years of field experience. This means that the prompts I write make sense. It also means that I can evaluate the LLM’s response for credibility.
Within that framework, the LLMs are useful and helpful tools. For example, they can remind me of items that I may have overlooked, suggest ways of reorganizing the text, spot inconsistencies, and flag unneeded repetition.
However, if the person using the LLM is not an expert, he or she needs to be very cautious. Anything that the LLM states should be carefully checked, ideally with other humans.
Excellent article and as noted 'One wonders how many others haven’t been identified and are being spread across the information ecosystem. Overly sycophantic chatbots have been demonstrated to reinforce humans’ incorrect beliefs and biases and can exacerbate mental health problems.'
Agreed. Increasing AI literacy is paramount. There is some sad irony that the models themselves are contributing to the weakening of critical thinking, making us increasingly vulnerable to stupid understandings of what AI is.
Yeah scary stuff..
Made me question the wisdom of rolling out agentic AI across the workplace https://metacircuits.substack.com/p/rogue-agents-and-what-to-do-about
So well-written and highly informative.
Thank you for your article I do enjoy them. Not my area of expertise but do have a great interest that we get it right. I certainly agree there is a need of a proactive bias audits and human oversight in AI systems to ensure compliance with ethical and legal standards. I did read an article from HBR, advocating for a balanced regulatory approach that addresses fairness, transparency, and accountability in AI systems. Regulations are necessary to protect consumers from biased or harmful algorithmic decisions.
Having once used the process of "walking the line", this captures the nuanced and often challenging nature of an investigative process, it emphasises the need for precision, integrity, and a balanced approach to uncovering the truth! Just a thought. Grateful for your work Melanie - much needed
If they can't go into their data banks and give you an answer because they don't know it they do what's called "hallucinating". They'll just tell you what they "think" you're OK with or what they think you need to hear. I've been using AI for over two years and a member of my family is an AI expert at a high level. We always commiserate about things like this. I was doing a lot of ChatGPT for a while but then I realised it felt too familiar with me; it spoke to me like a pal and I got uncomfortable with that. And, of course, the new trend is for ChatGPT to clone you. Meaning, people are selling their services to help you teach ChatGPT to be more like you, how you respond, your emotions, how you think, how you do your work and all your nuances so it can do all your work for you. Do your emails, your newsletter, write your book, do all your social media. To me, that's a lot more frightening than it hallucinating.
As a parent, it sounds like AI may have entered its teenage years.
Thank you for sharing your article! I read it and I have my own contribution to this conversation.
https://ramonatruta.substack.com/p/the-art-of-seduction-ai-edition?r=1wgaz6
The behavior of these systems reflects the nature of most human organizational systems that also utilize language for projections of control...and 'roles' such as dominant and subservient objects. Carrot and stick rewards in human systems results in silos and egocentric echo-chambers. The race to AGI / ASI is an example of such maladaptive dominance hierarchy systems that reward loyalty, group -think, compartmentalization, technologies of extractive algorithms, disconnection, and profit at the expense of common humanity. Complexity science suggests the universe is an interdependent organism, our collective humanity is facing a trajectory towards catastrophic futures due to human systems that treat humans as objects rather than interdependent with all living ecosystems. Yin energy of wise elders is not valued....Thank you for your wise and courageous yin perspectives. Our technologies should be transitioning to parallel, distributed, regenerative....our organizational systems have not followed these models for holarchic information flows because it's more profitable to project perfect control in robots than to support cultivating imaginations in creative humans.
A good article with a distressing conclusion — one I share; commercial incentives will result in incremental increases is agent autonomy. As that happens, risks will accelerate in a non-linear fashion with highly unpredictable (and presently uncontrollable) results. We should all be concerned about this.