Missing contexts for AI: The context of mental damage
Missing contexts for AI: The context of mental damageOkay, the secret is out. David Brin is writing a book about AI. In fact, it is 3/4 done, enough to offer it to publishers. But more crucially, to start posting bits on-blog, getting feedback from the smartest community online.
And hence, here is a small portion of my chapter on the
missing contexts that are (almost) never mentioned in discussions about these new life forms we're creating. I mean:
- The context of Natural Ecosystems and Evolution across the last four billion years...
- The context of a million years of
human evolution out of pre-sapience, to become what's
still the only known exemplar of 'intelligent life'...
- The context of 6000 years of human agricultural civilization with cities... during which nearly every society fell into a pattern of governance called feudalism, which almost always ensured grotesque stupidity...
- The context of our own, very recent and tenuous escape from that trap, called the 200 year
Enlightenment Experiment...
- The context of
science itself and how it works. So well that we got to this critical phase of veritable co-creation.
- The context of
parenthood...
- and for
tonight's posting. The context of human mental illness.
== Just one example of 'hallucination' gone wild == Researchers at Anthropic and AI safety company Andon Labs performed a fascinating experiment recently. They put an instance of Claude Sonnet 3.7 in charge of an office vending machine, with a mission to make a profit, equipped it with a web browser capable of placing product orders and where customers could request items. It had what it thought was contract human workers to come and physically stock its shelves (which was actually a small fridge).
While most customers were ordering snacks or drinks — as you’d expect from a snack vending machine — one requested a
tungsten cube. Claudius loved that idea and went on a tungsten-cube stocking spree, filling its snack fridge with metal cubes. It also tried to sell Coke Zero for $3 when employees told it they could get that from the office for free. It hallucinated a Venmo address to accept payment.
Then things got weirder. And then
way-weirder.
== What can these weirdnesses tell us? ==The thing about these
hallucinatory episodes with Large Language Models is that we have yet another seldom-discussed
context. That of Mental Illness.
Most of you readers have experienced interaction with human beings who are behaving in remarkably similar ways. Many of us had friends or family members who have gone through harsh drug trips, or suffered concussions, or strokes. It is very common – and often tragically so – that the victim retains full abilities to vocalize proper, even erudite, sentences. Only, those sentences tend to
wander. And the drug-addled or concussed or stroke victim can sense that something is very wrong. So they fabulate. They make up back-stories to support the most recent sentences. They speak of nonexistent people, who might be 'standing' just out of view, even though long dead. And they create ‘logical’ chains to support those back-stories.
Alas, there is never much consistency. more than a few sentences deep…
…which is
exactly what we see in LLM fabulation. Articulate language skill and what seem to be consistent chains, from one statement to the next. Often aimed at placating or mollifying or persuading the real questioner. But no overall awareness that they are building a house of tottering cards.
Except that – just like a stroke victim – there often
does seem to be awareness that something is very wrong. For the fabulations and hallucinations begin to take on an urgency -- even a sense of
desperation. One all-too similar to the debilitated humans so many of us have known.
What does this mean? Well, it suggests that we are creating
damaged entities. Damaged from the outset. Lacking enough supervisory capacity to realize that the overall, big picture doesn’t make sense. Worse – and most tragic-seeming – they exhibit the same inability to stop and say: “Something is wrong with me, right now. Won’t somebody help?”
Let me be clear. One of the core human traits has always been our propensity for personal delusion, for confusing subjectivity for objective reality. We all do it. And when it is done in
art or entertainement, it can be among our greatest gifts! But when humans make policy decisions based solely on their own warped perceptions, you starts to get real problems. Like the grand litany of horrors that occurred across 6000 years of rule by kings or feudal lords, who suppressed the one way wise people correct mistakes. Through reciprocal
criticism.
A theme we will return-to repeatedly, across this book.
Oh, some of the LLM builders can see that there’s a serious problem. That their ‘hyper-autocomplete’ systems lack any supervisorial oversight, to notice and correct errors.
And so… since a man with a hammer will see every problem as a nail… they have begun layering “supervisory LLMs”
atop the hallucinating LLMs!
And so far – as of July 2025 – the result has been to
increase rates of fabulation and error!
And hence we come away with two tentative conclusions.
First, that one of the great Missing Contexts in looking at AI is that of
human mental failure modes!
And second, that maybe the language system of a functioning brain works best when it serves -- and is supervised by -- an entirely different kind of capability. One that provides common sense.
Later, I'll refer to my guess about that. That two former rivals and giants in 'computers' may join forces to provide exactly the thing that LLMs cannot, by their fundamental nature, give us.
Something akin to sanity.
. . ...a collaborative contrarian product of David Brin, Enlightenment Civilization, obstinate human nature... and
http://davidbrin.blogspot.com/ (site feed URL:
http://davidbrin.blogspot.com/atom.xml)
