When the Signs of Intelligence Don’t Add Up
Why fluency, reasoning, judgment, and reliability may not travel together
There are moments when an AI system can appear remarkably intelligent.
It can analyze a difficult argument, write useful code, identify patterns across a large body of material, or make an unexpected connection between ideas.
Then, sometimes in the next exchange, it does something astonishingly foolish.
It misreads a simple instruction. It invents a source. It overlooks an obvious contradiction. It expresses confidence in a conclusion that falls apart under basic scrutiny.
The usual explanation is that the systems are still immature. As they improve, the foolish mistakes will diminish and their performance will become more consistent.
That may be true. But another possibility deserves attention. Perhaps the real problem lies not only in the systems, but in the way we interpret them.
We have learned to treat fluency, reasoning, understanding, judgment, and reliability as signs of a single underlying faculty. When these qualities appear together in a person, we call that person intelligent.
AI unsettles this familiar pattern. It can display some of the outward signs very strongly while other expected qualities remain weak, unstable, or unclear.
The signs are still there.
They simply no longer add up in the way we expect.
Sutton’s bitter lesson
Richard Sutton’s well-known essay, “The Bitter Lesson,” offers a useful starting point.
His argument was that the most important advances in AI came from general methods that could exploit increasing computation. Systems built around learning and search tended to outperform systems in which human experts tried to encode their own knowledge directly.
The lesson was bitter because it reduced the importance of carefully designed human expertise. Progress came from scale, learning, data, and computation, not by mimicking and encoding human-like intelligence.
Recent AI appears to follow this pattern.
Large models have acquired abilities that were not individually programmed into them. They can translate, summarize, write code, answer questions, and move between subjects with surprising ease.
Some of these abilities seem to appear once systems reach a certain scale.
The comparison with phase transitions in physics is tempting. Water warms gradually, but at a certain point it changes state. Something similar may happen in AI as it scales. Quantitative increases can produce what looks like a qualitative change.
Yet we do not fully understand these thresholds.
A new capability may genuinely appear. It may also have existed in weaker form before we noticed it. In some cases, the way we measure performance may make the change look more sudden than it really is.
The word “emergence” is used to describe what we observe. It does not always explain why it happened.
Still, the practical result is striking. These systems can now perform tasks that we once treated as evidence of intelligence.
The difficulty begins when we decide what that evidence means.
What does performance tell us?
Alan Turing famously moved away from the question of whether machines can think. He proposed a practical alternative. If a machine can behave in ways that appear intelligent, perhaps that is enough.
This was an important shift: it replaced a difficult question about inner experience with something we could observe.
Modern AI gives Turing’s position new force.
When a system explains a complex idea, responds to objections, adapts its language, and produces a fresh argument, it becomes hard to dismiss the performance as trivial.
Yet the same systems also expose the limits of behavior as evidence.
They can sound thoughtful without being dependable. They can produce convincing explanations without recognizing where those explanations are weak. They can follow the form of reasoning and still fail to notice when the reasoning has gone wrong.
Their performance encourages us to attribute intelligence; their inconsistency makes that attribution uncertain.
The issue is not simply whether an answer is true or false.
An answer may be factually wrong. It can also fail because it is incomplete, poorly framed, based on weak evidence, or inappropriate for the decision at hand.
A response can be technically accurate and still be misleading. It can answer the question asked while missing the more important question that should have been asked.
This is why the problem cannot be reduced to truth alone. The deeper issue concerns how well the answer relates to evidence, context, relevance, and judgment.
The old question of understanding
John Searle’s Chinese Room argument still matters here.
In simplified form, Searle argued that a system could manipulate symbols correctly, given the proper instructions, without understanding what those symbols meant.
The argument has been challenged many times. One important objection is that even if a single part of a system does not understand, the larger system may still do so.
Even so, Searle identified a problem that remains unresolved: producing an appropriate response does not by itself prove that understanding is present.
Generative AI makes this question harder because its responses are flexible, contextual, and often surprising. These systems do more than retrieve fixed answers. They generate, combine, infer, and adapt. That does not prove that they understand, but it makes the claim that they are simply manipulating symbols less satisfying than it once was.
The question remains open.
Hinton’s challenge
Geoffrey Hinton takes a different view.
He has argued that it may be too easy to dismiss neural networks as systems that simply match patterns. They learn distributed representations. These representations capture relationships among words, concepts, objects, and situations. They can then use those relationships across different tasks.
Perhaps this deserves to be called a form of understanding.
The word “merely” often does too much work in discussions of AI. Human beings also learn through patterns, prediction, association, and memory. Describing AI as statistical does not settle the matter.
The harder issue is what we mean by understanding.
Is it the ability to use concepts flexibly? Does it require direct contact with the physical world? Must it involve consciousness? Does it depend on having stable beliefs? Must a system recognize its own uncertainty?
There is no settled answer.
This takes us into both epistemology and ontology. How could we know that a system understands? And what kind of entity would it need to be for understanding to belong to it?
Get Nicos Rossides’s stories in your inbox
Join Medium for free to get updates from this writer.
A short article cannot settle these questions, but AI has made them increasingly difficult to avoid.
The human template
Part of the confusion comes from the way the concept of intelligence developed. We formed it mainly by observing other human beings.
In human life, fluent language, reasoning, memory, practical competence, understanding, and judgment often appear together within the same person.
Not always, of course. People can be articulate and mistaken. Knowledgeable and unwise. Clever and reckless. Even so, the qualities usually appear within a recognizable human whole. We therefore came to treat them as expressions of one underlying faculty.
AI presents a different case.
It can show fluency without stable belief, produce reasoning-like answers without reliable self-correction, and move across many domains yet still fail on a simple variation. It can generate something novel without having any real intention or sound confident without having good grounds for that confidence.
This may explain why AI can appear astonishingly intelligent and astonishingly foolish in the same conversation. The apparent contradiction could lie partly in the categories we bring to the system. We assume that when one familiar sign appears, several others must be present as well. AI suggests that this inference is no longer safe.
There is a further difference in how these capacities are acquired. A person who performs a task well has usually learned from doing it, and carries that learning into the next attempt. Most current systems do not, at least for now. They draw on far more text than anyone could read in a lifetime, yet they retain little from one working session to the next. In their case, competence has been separated from the experience that would normally produce it.
AI can generate a flood of plausible output; verification and judgment remain selective.
What is really coming apart?
It would be too strong to say that intelligence itself is coming apart. That would assume that intelligence is a settled and unified thing. We do not know that.
What is coming under pressure is our inherited concept of intelligence.
The term bundles together several capacities: language, memory, reasoning, adaptation, understanding, judgment, self-correction, and the ability to deal responsibly with uncertainty, just to name a few. We often speak as though these capacities naturally belong together. AI raises the possibility that they can appear in very different combinations.
This does not prove that intelligence is innately fragmented in some philosophical sense. It tells us something more modest, that the label may be too broad to explain what we are observing.
Calling a system intelligent can hide more than it reveals.
Beyond true and false
The most serious concern is often described as reliability. But even that word needs care. We do not simply need answers that are factually correct.
We need answers that are sufficiently complete, relevant, appropriately framed, and open to correction.
Consider a business decision.
An AI system may produce an accurate market analysis but ignore a regulatory constraint. It may summarize the available evidence correctly but fail to notice that the evidence is weak, or answer the question it was given while leaving the real problem untouched. It may generate a plausible strategic recommendation without understanding the political, cultural, or organizational conditions required to implement it.
None of these failures can be reduced to a simple true-or-false test. The issue is whether the output is adequate for the purpose it is meant to serve. That requires more than factual accuracy.
It requires context, framing, judgment, and an awareness of what may be missing.
The problem of the corpus
Part of the difficulty lies in the material on which these systems are trained.
A language model learns from human-produced content. That includes a wildly diverse set of sources: scientific knowledge, historical records, journalism, opinion, advertising, and propaganda, including all the associated human prejudice, repetition, and error.
The volume is enormous. But volume does not resolve the problem.
Prominence or prevalence does not guarantee a source lacks distorting bias, or even that it is accurate. A false claim may be widely repeated. A correct claim may remain obscure. A respected consensus may later collapse. A marginal view may eventually prove right.
The model learns patterns in what human beings have written. It does not encounter reality in a direct and uncontested form.
It can compare sources, detect inconsistencies, weigh evidence, and produce a plausible synthesis. But where the underlying body of knowledge is incomplete or mistaken, the model may simply reproduce the problem in a more polished form.
It can distil the corpus, but it cannot guarantee that the corpus is sound.
At some point, real knowledge requires contact with the world through observation, prediction, experiment, measurement, intervention, or consequence. Even then, interpretation remains necessary; evidence does not explain itself.
Why this matters in organizations
These questions are not only philosophical. Organizations are already deploying AI in their workflows, using it to prepare reports, analyze customers, write software, and even to support decision-making and generate strategic options.
The gains are real.
Many forms of cognitive work can now be produced more quickly and at lower cost. But the greater volume of plausible output creates another burden.
Someone still has to decide whether the question was framed correctly, whether the evidence is relevant, whether the assumptions are defensible, and whether the answer fits the situation. Someone also has to notice what is missing. And someone must accept responsibility when the conclusion is wrong.
The main danger is not simply that AI will make mistakes.
The more troubling possibility is that it will produce outputs that look polished, complete, and ready for use even when they are incomplete, poorly framed, or unsuited to the context.
This is why judgment becomes more important as AI improves. The systems may become better at generating analysis while remaining uneven at recognizing the limits of that analysis. The bottleneck then shifts from production to evaluation.
A better set of questions
Discussion of AI often returns to a simple question:
Is it intelligent?
The question may now be too blunt. It assumes that intelligence is one thing and that a system either possesses it or does not.
A more useful approach is to ask which capacities are present, under what conditions they appear, how far they generalize, what kinds of error they produce, what they fail to notice, and how well they deal with uncertainty, context, and framing. These questions are less dramatic, but they are more revealing.
AI may not be moving toward human intelligence along a single line. It may be developing a different cognitive profile, one in which some capacities become very powerful while others remain fragile, intermittent, or unclear.
That does not prove that AI is intelligent, nor that it lacks understanding. It tells us that the word itself may no longer be doing enough analytical work.
The deeper lesson is not that intelligence has literally come apart. It is that the signs we once used to recognize it no longer lead reliably to the conclusions we once drew.
AI has made the label harder to use casually.
That may be one of its most important contributions to our understanding of intelligence.
Dr. Nicos Rossides is a CEO, educator, and author whose work spans management, education, and artificial intelligence. He is Co-Founder and Chairman of the Advisory Board of listening247, a global insights firm, and Professor of Management at Minjiang University’s International Digital Economy College. His Routledge titles include Engaging the Workforce (2022), Employee Engagement in Startups (2025), and AI-Powered Insight (2026, with Michalis A. Michael).
Support writers. All our nonprofit’s offerings here.
