Software Development Where AI Is Useless
Introduction
Handing a coding agent everything from implementation to tests to the pull request is no longer unusual. Implementation productivity has risen to the point where one is tempted to say there is no code AI cannot write.
And yet the field still contains work that clearly cannot be handed to AI. Interestingly, this work is not necessarily “hard.” Much of it — a one-line configuration change, a single word of approval — is simple as a task.
This article makes one claim: the places where AI is useless are determined not by any shortfall in AI’s capabilities, but by structural boundaries inherent in the practice of software development. There are five such boundaries, and two axes that sharpen them. Once you understand them, you can decide where to deploy AI and where to keep people based on principle rather than intuition. And at the end, we arrive at a slightly surprising conclusion: the work of shrinking the places where AI is useless is precisely the work where AI is useless.
Chapter 1: Separating “Not Yet” from “Not in Principle”
Most articles arguing that “AI can’t do X” have gone stale with each model update. Hallucination, context-window limits, poorly calibrated confidence — these are limits of capability, and technical progress can ease them. This article does not deal with them.
What it deals with are limits of structure: limits that do not disappear no matter how capable the models become. They fall into five categories.
- The observation boundary. AI can only work within the context it is given. Information that has not been digitized might as well not exist.
- The correctness boundary. “What counts as correct” is a decision, not an inference, and AI does not hold that authority.
- The accountability boundary. Responsibility cannot be transferred to AI. An “approval” from an entity that cannot be held responsible does not function as approval.
- The independence boundary. Defense in depth assumes that failures are independent, but the errors of different AIs are correlated.
- The verification asymmetry boundary. AI’s value lies in generation speed. Where verification costs more than generation, that speed cannot be converted into value.
Two axes cut across these boundaries and sharpen them: the time axis (the consequences of a decision arrive years later) and the world outside the software (the user’s location, language, culture, and organization). Chapters 7 and 8 cover them.
Chapter 2: The Observation Boundary — What Isn’t Written Down Doesn’t Exist
An LLM-based agent is, at bottom, a function that takes context as input and returns output. Information not contained in the repository, the documentation, the logs, or the tickets is information the AI cannot know. And much of the information that matters most in software development is written down nowhere.
What decides the design lives outside the repository
As Conway’s Law suggests, most design decisions are shaped by organizational circumstances. Which team owns what, next year’s hiring plan, a key person’s likely departure, the balance of power between departments. These are constraints that shape the architecture, and none of them are in the repository. So an AI’s proposal can be technically correct and organizationally impossible. The physical world is the same: a timing problem visible only on an oscilloscope, or a failure that reproduces only in one customer’s environment, stays outside the AI’s field of view until someone turns it into text.
“When the tools die”: AI is most powerless exactly when it is most needed
The point this chapter most wants to stress is a paradox of incident response.
The ceiling on an AI agent’s capability is the availability of its tools. It can assess the situation because the monitoring is visible, obtain credentials because the identity provider is up, and reach the API because the network is connected. But in the truly hard incidents, those tools themselves are caught in the failure. Monitoring goes dark. The identity provider goes down and the AI can log in nowhere. The network partitions and the AI’s own API is unreachable. The harder the incident, the more likely the tools that support the AI are damaged along with it — so the difficulty of an incident and the usefulness of AI are inversely correlated.
Humans, by contrast, have out-of-band paths: the telephone, the physical console, badge access to the data center. Reseating a cable, checking a blinking LED. For now, only humans have these means of observation.
Lost context cannot be recovered
“Nobody knows why this strange code exists, but deleting it seems likely to break something.” This is Chesterton’s Fence. AI is good at the archaeology of git blame, but it cannot recover decisions that were never recorded—say, a verbal agreement with the sales director at the time. The same applies when the only person who knows that status = 7 means "deleted, but retained for legal reasons" is about to retire. Without a data dictionary, that meaning lies outside the observation boundary.
So does what people really think. When the true cause is political, people do not say so anywhere a record is kept. That “the CTO’s ‘interesting’ actually meant no” can be read only by someone who knows the history of the relationship.
The lesson of the observation boundary is simple: how useful AI is in your organization is determined by how much your organization has written down.
Chapter 3: The Correctness Boundary — “Correct” Is a Decision, Not an Inference
AI can write code that is correct against a given specification. But whether the specification itself is correct cannot be derived from the code or from the existing spec. What counts as correct is not the result of inference; it is someone’s decision.
The test oracle problem
Deciding a test’s expected value is, in effect, writing the specification. Ask AI to write tests and you tend to get snapshots of current behavior — tests that do not detect bugs but enshrine existing bugs as specification. Whether 0.1 + 0.2 yielding 0.30000000000000004 is a bug or the spec looks like a technical question, but it is a product decision.
Users do not know what they want until they see it
AI builds a prototype in minutes and dramatically speeds up iteration. But it cannot stand in for the experience of the user who looks at it and says “not this” or “yes, this.” Preferences are not sitting fully formed in someone’s head waiting to be extracted; often they take shape only through experience. This is the realm of tacit knowledge that the philosopher Michael Polanyi described as “we know more than we can tell.”
The same holds for subjective quality, such as a UI that feels slightly off, and for the handling of dirty data, such as thirty thousand records with a birth date of January 1, 1900. Someone has to feel; someone has to set the rule. AI can enumerate the options, but it cannot choose.
The lesson of the correctness boundary: AI can derive the “right answer” against a given standard. It cannot decide what counts as right.
Chapter 4: The Accountability Boundary — Responsibility Cannot Be Transferred
This is the most important boundary in the article. Much of software development consists not of “work” but of accepting responsibility.
Approval means accepting responsibility
The decision not to build something. The go/no-go call on a release. Accepting a risk: “we will not fix this vulnerability this quarter.” These look like matters of judgment, but at bottom they are declarations: “I will answer for the outcome.” AI cannot be sued, fired, or paged at 3 a.m. So an AI “approval” does not function as approval.
Tanaka’s LGTM means something not only because Tanaka’s judgment is good, but because for five years Tanaka has been woken at night by code they approved and has borne the consequences. Trust accumulates through a record of accepted responsibility — what Nassim Taleb calls skin in the game. Estimates are the same. In a plan, an estimate is not a prediction but a promise, and an estimate from someone who cannot be held to the promise is no foundation for a plan, however accurate.
Regulation demands a named human
DO-178C for airborne software, IEC 62304 for medical device software, ISO 26262 for automotive, SOX and J-SOX for financial reporting. These require the involvement of named humans: sign-off by a responsible person, traceability, segregation of duties. The requirement has nothing to do with AI’s accuracy, and it will not move until the standards and laws are rewritten. Giving AI deployment rights to production can itself be a control violation under segregation of duties. “Just give the AI the authority” does not transfer responsibility; it only blurs where responsibility sits. What an audit asks is: who did this?
Powerless by design
Keeping human approval as a bottleneck is not necessarily a process defect. Sometimes it is the very purpose of the control. Here AI is not merely powerless; it should be designed to be powerless.
The lesson of the accountability boundary: you can hand AI authority, but you cannot hand it responsibility.
Chapter 5: The Independence Boundary — AI Reviewing AI Is Not Defense in Depth
“Have a second AI review the first AI’s code and you’re safe” is intuitive, and it does help to a degree. But calling it defense in depth runs into an experiment from about forty years ago.
The Knight and Leveson experiment
In 1986, John Knight and Nancy Leveson compared 27 programs written independently from the same specification. The prevailing belief was that if independently developed programs were run under majority voting, their failures would be independent too, and reliability would rise dramatically — N-version programming. The results refuted that premise. The programs failed simultaneously on the same inputs far more often than the independence assumption predicted. Hard problems are hard for everyone, and everyone gets the tricky parts wrong.
If even human failures correlate, then AIs trained on similar corpora, shaped by similar RLHF, and optimized for the same benchmarks can correlate even more strongly. AI review of AI-generated code is less defense in depth than looking twice with the same eyes. Even combining models from different vendors yields less independence than it appears, given the overlap in training data — and it is a different kind of independence from that of a human reviewer, who fails in different ways and answers for the result.
Cognition itself becomes the attack surface
An AI agent may follow instructions embedded in the text it reads: prompt injection. This becomes critical wherever the agent processes untrusted input while holding privileges — external pull requests, bug reports, logs sent in by a customer.
In the 2024 xz Utils incident, the attack — two years of building trust to obtain maintainer rights — unfolded outside the code. The unease that “this contributor is oddly eager” arises only from long memory of the interactions and from observation of human relationships that never make it into the record.
The holder of the second key
For irreversible operations — dropping a table, rotating an encryption key — the essence of the human role is not judgment. Just as launching a nuclear missile requires two keys, a critical operation needs a second key holder independent of the operator. The human’s value here is not being smarter than the AI; it is being an independent party that cannot be moved by a prompt.
The lesson of the independence boundary: redundancy has value only when failures are independent of one another.
Chapter 6: The Verification Asymmetry Boundary — Fast to Build, Slow to Verify
Most of AI’s value lies in generation speed. But if the time it takes to verify the output does not change, overall development speed is capped by verification.
Where verification costs more than generation
Cryptography, financial calculations, lock design in concurrent code, permission checks. None are hard to write, but confirming they are truly correct is extremely expensive. If AI writes them ten times faster, the reviewer’s verification time does not shrink; only the review queue grows, and “verification debt” piles up. For irreversible operations the asymmetry reaches its limit: if the loss is unrecoverable, no failure probability is small enough. What is asked of the human is not speed but slowness — deliberate friction.
The economics of oracles
Verification is expensive, in most cases, because there is no oracle: no mechanical way to judge correctness. Conversely, where cheap and independent oracles exist — formal verification, property-based tests, contract tests, invariants expressed in types — AI becomes useful all at once. AI generates, the machine verifies, and the human only reviews the result.
Get Shibui Yusuke’s stories in your inbox
Join Medium for free to get updates from this writer.
In other words, the map of where AI is powerless is a map of where your organization lacks oracles, and therefore a map of where to invest.
To verify, you have to ship. And once you ship, you can’t go back.
Product engineering has one more, thornier kind of verification: does this feature have value? The correctness of code can, ideally, be checked before release. The only oracle for the value of a feature is actual use by the market and its users. To verify, you have to ship — and the moment you ship, the thing you were trying to verify changes.
The typical arc: you release a feature on a hypothesis. Usage comes in at 30% of expectations. The data says rebuild it or remove it. But a small number of users have started relying on it daily, and support hears that “our operations stop without this feature.” Change it and you provoke a backlash; remove it and you lose trust; leave it and maintenance cost just accumulates.
As Hyrum’s Law tells us, with enough users, someone will depend on every observable behavior of your system regardless of what the contract promises. A feature shipped as an experiment becomes a fait accompli regardless of the experiment’s result. Feature flags and staged rollouts soften this, but some verification results can only be obtained through a period of real use, so they cannot eliminate it.
The decision not to decide yet
The feature enters limbo: not successful enough to keep investing in, not failed enough to remove. What is called for is not a clear answer but a delicate decision. “Leave it alone this quarter and reassess next quarter.” “Hide it from new users, keep it for existing ones.” All are feasible, and the data cannot say which is right.
Worse, not deciding yet is itself a decision — and often the right one. The optimum shifts with the business environment six months out, so deferring the decision to preserve options is rational behavior under uncertainty. But deferral needs a deadline and an owner. A deferral with no answer to “who will re-decide, by when, and based on what” is not a decision; it is neglect.
AI can lay out the options and estimate their impact. It cannot decide “not now,” and it cannot be there when the deadline arrives to decide again. This decision straddles the accountability boundary and the time axis of the next chapter.
The lesson of the verification asymmetry boundary: to make AI useful, invest in oracles. But the market is the one oracle you cannot build cheaply, and there what counts is not speed of verification but the ability to decide, and to manage deferral.
Chapter 7: The Time Axis — AI Is Not There “When the Time Comes”
The five boundaries share a common thread: time. AI processes the context of “now, this moment,” but software development is a practice that runs for years. The consequences of a decision come back years later. AI can read the past as a record, but it has not lived through that time. And it is not there when the consequences arrive.
“When” is not written in the repository
The same change can be right in March and wrong in April. Don’t touch the accounting code right before quarter close. Don’t change the UI the week before a campaign. A key member leaves next month. A competitor shipped a new feature last week. All of these govern what should be done now, and almost none of them are in the repository or the tickets. Take the custom of “no deploys on Friday evening”: the reason behind it — few people available on weekends — lives in the calendar and the organization, not in the code.
The consequences come back years later
A major design decision is a bet on the future of the business. A bet needs a bettor who is still there when it pays off, to take the win or the loss. Whether the database choice or the move to multitenancy was right becomes clear three or five years later. An organization learns by absorbing those consequences, and that learning is available only to someone present at both the decision and its outcome. AI can lay the two records side by side and analyze them, but the continuity of living through the interval, absorbing the result, and applying it next time does not exist inside a session. The same goes for deciding when to repay technical debt, and for the “not yet” decision of Chapter 6: both hold only if there is someone to decide again when the deadline comes.
Trust and unease are both functions of time
Tanaka’s LGTM was the product of a five-year track record; the xz Utils attack could not have been noticed without two years of observation. AI can be given memory, but holding the past as a record is not the same as spending that time in the field. The sense that “something is off with that person lately” arises only in someone who knew them before the change, and the one who acts on that sense and answers for it is, again, an entity that persists across time.
The lesson of the time axis: AI is not there at the moment of decision, nor at the moment the consequences arrive. Someone has to remain in the field across time, absorb the results, and decide again. For now, that is people — and organizations made of people.
Chapter 8: Outside the Software — Users Live on the Other Side of the Screen
The largest body of unwritten information lies outside the development organization. Software is used in the real world. Users are in a specific place, speak a specific language, and live inside a specific culture and organization. The AI’s context can take in the software and its records; it cannot take in the world where the software is used.
Place: where is the user using it?
An inventory app used at the back of a warehouse where the Wi-Fi keeps dropping. A terminal operated in gloves inside a freezer. A screen viewed in direct sunlight. Assumptions like “responses return in 200 milliseconds” or “a 44-point tap target is enough” are overturned by where the user is. The requirement of offline-first does not come from the code; it comes from the field. Place also has legal meaning. Under privacy laws such as Japan’s APPI, the GDPR, and export-control regimes, the “correct storage configuration” is determined by the user’s location and jurisdiction.
Language: the right answer is not in the string
Variant kanji glyphs, the wave-dash encoding problem, half-width katakana, hierarchical postal addresses, fiscal years and era names. AI knows these Japanese pitfalls as general knowledge. The hard part is what knowledge alone cannot solve. Sorting names in Japanese phonetic (gojūon) order requires the reading, but the string 山田 (Yamada) does not contain its own pronunciation. The right answer lies outside the data — which is why Japanese forms have a separate field for the reading. Likewise, whether an error message should say nyūryoku shite kudasai or the more deferential go-nyūryoku kudasai (both “please enter”) is decided not by grammar but by the relationship between the product and its users.
Culture and organization: the same symbol means the opposite
In Japanese and Chinese stock-trading apps, rising prices are shown in red and falling prices in green or blue — the reverse of Western convention. In Japanese schools, correct answers are marked with a circle and wrong ones with an × or a check mark, so a check does not necessarily mean “done.” Culture extends to how work gets done: ringi approval circulation, hanko seals, month-end closing with payment at the end of the following month, year-end budget spending. Software that does not fit these customs goes unused no matter how good it is technically. And customs the participants take for granted are never written down.
Users have organizations too. Hospital shift systems, the cost of a stopped factory line, who decides the purchase and who opposes it. “We can’t remove that; it’s the client’s department head’s pet feature” makes the limbo of Chapter 6 even harder to escape. AI knows “Japan in general” and “hospitals in general” very well. But the specific world your users live in is unknowable unless someone goes to the field, observes, and brings it back. AI cannot go to the field.
The lesson of the world outside: the correctness of software is defined outside the software. What AI knows is the general world; bringing back the specific world your users live in is human work.
Chapter 9: Where AI Is Useless, Stage by Stage
Here are the five boundaries and two axes laid out along the development lifecycle. Parentheses give the relevant boundary or axis.
- Requirements and planning: deciding not to build (accountability) / the experience in which preferences form (correctness) / going to the field and bringing back place, language, culture, and organization (observation, outside)
- Design: structural decisions that depend on team composition and hiring plans (observation) / irreversible technology choices that bet on the future (accountability, time) / data-residency decisions by jurisdiction (outside, accountability)
- Implementation: bugs rooted in the physical environment (observation) / integrating with internal systems that have neither documentation nor access (observation)
- Review: judging not “is it correct” but “should this exist” (observation, accountability) / independent eyes on AI-generated code, and handling untrusted input (independence)
- Testing: deciding expected values (correctness) / judging culture-dependent meaning (correctness, outside)
- Release: go/no-go and “when” (accountability, time) / regulatory sign-off and segregation of duties (accountability) / the fate of a feature in limbo (verification asymmetry, accountability, time)
- Operations and incidents: incidents where the tools themselves are down (observation) / incident command (accountability) / the second key for irreversible operations (independence, verification asymmetry)
- Security: accepting risk (accountability) / the premise of “who is the adversary” (observation) / excluding AI from regulated data (accountability)
- Data and migration: interpreting data whose meaning exists only in someone’s head (observation) / setting rules for dirty data (correctness)
- Maintenance: recovering unrecorded decisions (observation) / timing debt repayment and managing deadlines for deferred decisions (time, accountability)
Laid out this way, the work where AI is useless is distributed not by difficulty but along the structural boundaries and the two axes.
Chapter 10: The Boundaries Can Be Shrunk. The Core Remains.
By now this may sound like “AI has limits, so let’s protect the human sanctuary.” The conclusion of this article is the opposite.
Two of the five boundaries, and parts of both axes, can be shrunk by organizational effort.
- The observation boundary shrinks with investment in documentation and observability: record the reasons for decisions as ADRs (Architecture Decision Records), build a data dictionary, and provide a path to situational awareness even when the main tools are down.
- The verification asymmetry boundary shrinks by building oracles: express invariants in types, put property-based and contract tests in place, and introduce formal verification where it applies.
- The world outside shrinks by bringing it in: do user research and field observation, and document personas and locale requirements. The act of bringing it back, though, happens only when a person goes to the field.
- The time axis can be partly supplemented by records: write review dates into ADRs, attach a deadline and an owner to every deferred decision, and share quarter closes and contract renewals as a calendar. But the entity that is present when that date arrives and decides again cannot be replaced by a record.
What remains after shrinking is the correctness boundary, the accountability boundary, the independence boundary, and the entity that absorbs consequences across time. Deciding what counts as correct. Answering for the outcome. Standing as an independent pair of eyes. Being there when the time comes. These are not jobs that AI’s progress will erase; they are the core of software engineering itself.
Which brings us back to the paradox from the introduction. What to record in an ADR, which properties to write into a property-based test, what to look for in the field, when to set the deadline for a deferral. AI can write the documents and the test code, but deciding what should go into them sits on the correctness and accountability boundaries.
The work of shrinking the places where AI is useless is precisely the work where AI is useless.
In Closing
- Observation boundary: what isn’t written down doesn’t exist. Write it down and the boundary shrinks.
- Correctness boundary: correctness is a decision, not an inference. People decide.
- Accountability boundary: authority can be handed over; responsibility cannot. Human approval is also a control that keeps AI powerless on purpose.
- Independence boundary: redundancy counts only when failures are independent. A person holds the second key.
- Verification asymmetry boundary: fast to build, slow to verify. Invest in oracles; handle the market oracle by deciding and managing deferral.
- Time axis: AI is not there at the moment of decision, nor when the consequences arrive.
- Outside the software: correctness is defined outside. People bring back the specific world.
What coding agents changed is the “work” part. Deciding, answering for outcomes, standing as independent eyes, being there when the time comes, bringing the world back from the field — these remain human jobs. There are structural boundaries here; some can be shrunk, and what remains to the end is the core of the engineer’s work.
The next time you hand a task to an AI agent, ask yourself: which boundary does this work touch? When, where, and for whom is it being done?
