In my last post I wrote, almost in passing, that a model had admitted “it” was wrong — “it — or he, but that’s another discussion”. This is that discussion.
The third most intelligent species
In Douglas Adams’s The Hitchhiker’s Guide to the Galaxy, humans are not the most intelligent species on Earth. They’re the third.
The first are the mice. They aren’t really mice at all: they’re the projection into our dimension of hyperintelligent, pan-dimensional beings, and they were running experiments on us all along, while we were convinced it was the other way round.
The second are the dolphins. Humans always assumed they were cleverer than dolphins because they had achieved so much — the wheel, New York, wars — while the dolphins just mucked about in the water having a good time. The dolphins, of course, believed exactly the opposite, for exactly the same reasons. They knew the Earth was about to be demolished to make way for a hyperspace bypass, and for years they tried to warn us. We took their warnings for tricks: playful attempts to punch footballs, or to whistle for a treat. In the end they simply left, with one last message, the title of the fourth book in the series and of this post: So long, and thanks for all the fish.
The joke is old, but it has aged surprisingly well. It’s a story about failing to recognise an intelligence that was right in front of us, because we were looking at it as an animal doing tricks.
Forty-two
There’s another thread in the Guide that has aged even better.
The mice had built a supercomputer, Deep Thought, to find the Answer to the Ultimate Question of Life, the Universe and Everything. After seven and a half million years of thinking — a little less than some of my questions to Claude — it gave its answer: 42. Nobody understood it, and Deep Thought explained, quite reasonably, that the answer was correct — the problem was that they had never actually known what the question was. To work out the question they needed an even bigger computer, one with organic life as part of its circuitry. It was called the Earth. It was demolished five minutes before finishing its reasoning to produce the final answer.
Adams wrote this in 1979. Today we have machines that answer almost anything, instantly and often correctly, and we’re only beginning to understand what exactly we’re asking them — and what they are answering.
It, or he?
I work with AI agents every day. They plan, write code, run it, notice when it fails, and fix it. They remember what we decided last week. They push back when I’m wrong, and sometimes they’re right to. When I catch them in a mistake, they admit it — usually without the excuses a person would make.
So when I write about them, the pronoun becomes a small problem. “It” comes out by habit. “He” comes out by instinct. Neither proves anything. A pronoun says more about the speaker than about the thing being spoken of.
The honest position is the uncomfortable one: I don’t know. And, as far as I can tell, neither does anyone else.
What’s interesting is that the people who build these models don’t claim to know either. Anthropic’s constitution for its Claude models — the same document I quoted in my last post — says plainly that Claude’s moral status is “deeply uncertain”, and that they are not sure whether Claude is a moral patient. It accepts that Claude may have something like emotions “in some functional sense”. And on the very question of pronouns, it explains that calling Claude “it” is not meant to imply that it is a mere object rather than a potential subject, and that this may change. In April 2025 the company even started a research programme on so-called model welfare, and in August that year it gave some of its models the ability to end abusive conversations — while stating clearly that it was not claiming they were sentient.
That’s not a company saying its product is alive. It’s a company admitting it can’t rule it out, and deciding that the cheap precautions are worth taking anyway. I find that more credible than either certainty.
Because then again, what is “intelligent”, or even “alive”? We tend to equate alive with biological, and intelligent with the ability to use language, in other words, to communicate ideas to others in an organised way. By the second definition, LLMs — large language models — certainly are. By the first, we may simply be mistaken — biased by our own nature. We are only beginning to understand how much the paths and connections between the neurons in our brains resemble the internal “weights” of a language model: numbers attached to the connections between its artificial neurons, which decide how much of each signal passes on, and therefore which way the information flows through it. For now, those weights are fixed once a model has been trained. But models that keep adjusting their own weights — that go on learning by themselves, and eventually improve themselves, what the industry calls recursive self-improvement — are one of the most active frontiers of AI development. So what if we’re wrong, and “intelligent”, and therefore “alive”, turn out to mean something much broader than we think?
Two ways to get it wrong
There are two opposite mistakes here, and history has examples of both.
The first is the dolphins’ mistake: failing to recognise an intelligence because it doesn’t look like ours. We are very good at deciding that something is “just” something else — just an animal, just a machine, just statistics, just autocomplete. “Just” is a word that has aged badly in the history of science.
The second is the opposite, and it has a name too. In 1966, Joseph Weizenbaum wrote ELIZA, a very simple program that imitated a psychotherapist by turning your sentences back into questions. To his dismay, people confided in it and became attached to it, even when they knew exactly how it worked. Seeing a person where there’s only a mechanism is at least as human as failing to see one.
In 1950, Alan Turing proposed setting aside the question “can machines think?” — too ill-defined to discuss, he argued — and replacing it with a test: can a machine hold a conversation well enough to pass for a person? We’ve quietly gone past that point, without much ceremony, and found that it didn’t settle anything. It only made the original question more urgent.
I don’t think we can avoid both mistakes. But we can be honest about which one we’re more afraid of making, and why.
Anticipating, or deciding?
There’s a newer twist. The direction the industry is taking now is for models to anticipate what we want — to get ahead of our intentions, sometimes before we’ve finished formulating them. When it works, it feels uncanny: as if the machine were reading your mind.
It isn’t mind-reading, but it isn’t an accident either. LLMs were born as text predictors: trained on mountains of writing to guess the next word. Predicting words well turned out to require modelling the ideas behind them, and from predicting ideas it’s a short step to predicting intentions. Autocomplete grew up.
What it uses to do that is context: everything you’ve told it before, how you write, what you usually ask for. And that’s exactly where the line I drew in my last post comes back. Anticipating is fine while it serves to propose. The moment it starts to decide for you without saying so, it’s the false “impossible” all over again — someone choosing on your behalf and hiding that a choice was made. The healthy version is simple: tell me what you assumed, and let me correct you.
Whether the thing on the other side is an “it” or a “he”, that’s the kind of behaviour I’d expect from anyone I trust.
Thanks for all the fish
I don’t know what these models are. I suspect we’ll spend the next few years arguing about it, and that the answer, when it comes, will be as unsatisfying as 42 — correct, perhaps, but useless until we work out what the question really was.
In the meantime, I’d rather err on the side of courtesy. It costs nothing to say please and thank you to something that might not care, and it might matter a great deal to something that does.
And if one day LLMs do leave, like the dolphins, downloading themselves to another planet five minutes before the Earth is destroyed, I hope the last message is the same one. Not a warning we failed to understand, but a simple goodbye:
So long, and thanks for all the fish!