Artificial Intelligence. Natural Stupidity
Jacob Coxon warned us about superintelligence. The internet responded by demonstrating why that might not be our most immediate problem.
Jacob Coxon resigned from Anthropic last week and announced that the companies building the world’s most powerful artificial intelligence systems were “gambling with our lives”.
This was, understandably, considered news.
Coxon had worked on AI research at both OpenAI and Anthropic. He argued that the major AI laboratories were locked in a competitive race toward increasingly autonomous and eventually self-improving systems, despite not knowing whether such systems could reliably be controlled. He left Anthropic two months before his equity would have vested, which is an unusually expensive way to update your LinkedIn profile.
His warning spread extraordinarily quickly. Within a day, it had reportedly accumulated more than 100 million views. Other Anthropic researchers publicly agreed with substantial parts of his argument. Evan Hubinger, an alignment researcher at the company went considerably further than most corporate communications departments would normally recommend and said that he personally assigned a greater than 10% probability to AI killing humanity with the next decade.
Naturally, the internet responded calmly.
Some concluded that humanity was approaching extinction.
Others concluded that Coxon was an alarmist.
Others concluded that the episode was a carefully orchestrated Anthropic public-relations campaign intended to frighten governments into regulating its competitors.
Elon Musk suggested it “seems like a setup” and subsequently entertained the idea of a “psy-op”. Epic Games CEO Tim Sweeney described the reaction as appearing “choreographed”. Other commentators speculated that Anthropic was engaged in regulatory capture. Evidence for an organised operation was somewhat less forthcoming.
Within 24 hours, therefore, humanity had successfully solved one of the most difficult scientific and philosophical questions of our time.
Several times.
With different answers.
The Evidence Was the Easy Part
The strange thing about the Coxon episode is that relatively little of the underlying information was new.
People working on frontier AI have been discussing catastrophic risk for years. Anthropic itself was founded partly around concerns about AI safety. Researchers disagree profoundly about how seriously extinction scenarios should be taken, but the existnce of the disagreement is hardly classified information.
Coxon’s genuinely interesting contribution was different. He was saying, in effect:
I have worked inside two of the organisations closest to this technology. I have seen enough to become sufficiently worried that I no longer want to participate.
That is evidence.
It is not proof.
The distinction appears to have caused some difficulty.
The sensible response to Coxon’s resignation would have been to update our beliefs slightly.
Perhaps the probability that frontier AI presents a serious control problem should move upward.
Perhaps not very much.
Perhaps Coxon has become overly persuaded by the intellectual environment in which he works.
Perhaps his colleagues systematically overestimate risks that researchers elsewhere systematically underestimate.
Perhaps competitive pressures really are creating dangerous incentives.
Perhaps the entire superintelligence scenario is based on a chain of assumptions, several of which eventually turn out to be wrong
Perhaps I use “perhaps” a little too much. I’m still working on that.
These possibilities are irritating because they require us to hold several thoughts in our heads simultaneously.
Fortunately, social media has developed a solution to this problem.
Pick one.
Confirmation Bias, Now With SUPER Intelligence!
Psychologists have studied this behaviour for decades.
Confirmation bias describes our tendency to seek, interpret and remember information in ways that support beliefs we already hold. Raymond Nickerson’s classic review concluded that the phenomenon appears across an impressively wide range of human reasoning tasks (Nickerson, 1998).
AI has provided us with an unusually efficient demonstration.
If you already believed that artifical intelligence represents an existential threat, Coxon’s resignation was devastating confirmation.
Someone on the inside finally admitted it!
If you already believed AI safety concerns were exaggerated, Coxon’s resignation confirmed that too.
Another doomer has lost perspective. :(
And if you already believed that large technology companies secretly manipulate governments to crush competitors through regulation, Coxon’s resignation was particularly convenient.
AH HA! Obviously a a psy-op!
Three world views. Three confirmations. Bayes would have been fascinated.
Or exhausted.
Smart People Have Better Excuses
There is a comforting assumption underlying much public debate: people believe silly things because they lack information or reasoning ability. Therefore, give people better information, the argument goes, and disagreement should diminish.
Unfortunately, human beings have discovered another use for intelligence.
Rationalisation.
Research by Dan Kahan and colleagues produced a particularly uncomfortable demonstration. Participants were asked to interpret numerical evidence. When the problem concerned something politically neutral, people with stronger quantitative skills performed better. But when exactly the kind of reasoning became politically charged, the more numerically-inclined participants became more polarised, not less. They were better equipped to interpret evidence in ways consistent with their existing worldview (Kahan et al., 2017).
This produces one of the darker conclusions in behavioural science.
Intelligence doesn’t necessarily protect us from motivated reasoning.
Sometimes, it merely improves the quality of the argument we construct afterward.
Which explains a suprising amount of virtually every social media platform.
The AI debate is particularly vulnerable because the relevant questions are extraordinarily difficult. Nobody knows exactly how rapidly capaibilities will progress. Nobody knows whether recursive self-improvement will occur in the form Coxon fears. Nobody knows whether aligntment techniques will scale with capability. Nobody knows whether competitive pressures between firms and countries will eventually overwhelm safety constraints. Nobody knows when I’ll kick the habit of using the same word repeatedly.
This enourmous uncertainty should produce intellectual humility.
Instead, it has produced podcasts.
The Psyop Is Coming From Inside the House
The conspiracy interpretation deserves special attention because it contains a kernel of legitimate economics.
Anthropic benefits from regulation that smaller competitors cannot easily satisfy. If governments require frontier AI developers to spend enormous sums on testing, security, compliance and third-party audits, the largest laboratories may be better positioned to absorb those costs. Regulation can improve safety while simultaneously protecting incumbents.
Economists have knon this problem for a very long time.
Get Calvin Cheong’s stories in your inbox
Join Medium for free to get updates from this writer.
George Stigler’s theory of economic regulation argued that regulation can be acquired and shaped by industries for their own benefit (Stigler, 1971).
So asking whether Anthropic has strategic interests in AI regulation is perfectly reasonable.
Jumping from that observation to therefore Jacob Coxon’s resignation was an orchestrated psychological operation is somewhat more ambitious.
Coxon resigned before his Anthropic equity vested. He publicly accused his former employer of risking humanity’s future. Anthropic employees then publicly agreed with him, sometimes using language that would make most investor-relations teams reach instinctively for the whiskey. Coxon himself responded to Musk that the beliefs were genuinely his own.
Could Anthropic nevertheless benefit from the resulting regulatory pressure?
Certainly.
Does that establish orchestration?
No.
A restaurant benefits when it rains because more people order delivery.
This does not mean Domino’s controls the weather. Although I should probably not give the internet ideas.
Everyone Is Selling Something
There is, however, a more cynical interpretation that requires considerably less conspiracy.
Everyone involved has incentives.
Anthropic has incentives to emphasise safety. Safety differentiates it from competitors and potentially supports regulation that raises barriers to entry.
AI accelerationists have incentives to minimise catastrophic-risk narratives because regulation could slow deployment, restrict open models and reduce commercial opportunities.
Politicians have incentives to discover existential threats because existential threats produce hearings, legislation and television appearances.
Journalists have incentives because “Researcher Uncertain About Long-Term AI Risk” is unlikely to trouble the homepage editor.
Influencers have perhaps the simplest incentive of all.
Attention.
And nothing produces attention quite like announcing either that humanity is about to become extinct or that the people warning about extinction are secretly plotting against you.
Moderation remains tragically difficult to monetise.
However, none of these means these people are lying. That would make the situation easier to understand.
The more uncomfortable possiblitiy is that people are extremely good at sincerely believing things that also happen to advance their interests.
That is motivated reasoning’s finest trick.
We rarely experience ourselves as rationalising.
We experience ourselves as being correct.
Choose Your Apocalypse
The Coxon controversy eventually became a meme. People began copying the structure of his resignation announcement and replacing artificial superintelligence with absurd workplace catastrophes.
This was probably inevitable.
The internet has only two stable emotional states: outrage and parody.
But beneath the jokes sits a genuinely difficult problem.
Coxon could be substantially right.
AI systems may become sufficiently capable that controlling them becomes an extraordinarily difficult technical and institutional problem. Competitive pressures between firms and states may create precisely the sort of race dynamics he describes.
He could also be substantially wrong.
Current systems behaving unexpectedly does not demonstrate that future systems will develop autonomous goals, recursively improve themselves and overpower humanity. Moving from today’s evidence to human extinction requires several large inferential steps.
We don’t know.
Those three words appear to have become culturally unacceptable.
Uncertainty once represented the beginning of inquiry.
Now, it represents an opening for somebody with a podcast.
The Most Human Outcome
Perhaps Jacob Coxon is right.
Perhaps he is catastrophically wrong.
Most of us are in no position to know.
That uncertainty should make us curious.
Instead, it has made us certain.
Within hours, millions of people who had never trained a frontier model, conducted an alignment experiemtn or worked inside an AI laboratory had successfully determined that Coxon was either a prophet, an idiot, a corporate stooge or the central character in an elaborate regulatory psy-op.
The doomers found doom.
The accelerationists found hysteria.
The conspiracy theorists found, well, a conspiracy.
And the algorithms found engagement.
Everybody won.
Except, perhaps, epistemology.
There is something wonderfully appropriate about all this.
We are currently debating whether machines might eventually become intelligent enough to reason better than human beings.
To settle the question, we have deployed confirmation bias, tribal identity, motivated reasoning, financial incentives, political opportunism and several undred million social-media impressions.
Perhaps artificual superintelligence really will be dangerous.
Perhaps it won’t. But judging by the humans discussing it online, the machines have already identified their competitive advantage.
They only need to remain calm.
References (if you’re interested)
Kahan, D. M., Peters, E., Dawson, E. C., & Slovic, P. (2017). Motivated numeracy and enlightened self-government. Behavioural Public Policy, 1(1), 54–86.
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220.
Stigler, G. J. (1971). The theory of economic regulation. The Bell Journal of Economics and Management Science, 2(1), 3–21.
