Same Model, 25% Better Code: Why Your AI Agent’s Harness Matters More Than the LLM
The idea in 30 seconds
An LLM on its own is a text generator: it takes an input and returns an output, with no memory between calls and no access to anything outside itself. To become an agent that reads files, runs code, and checks its own work, it needs a layer of software around it: the harness. And that layer matters. The same model, in different harnesses, can produce very different results.
To see why, think of a senior developer who has just joined the team. Without the repository, the terminal, and the tests, they can only describe what they would do. With that environment set up, the same developer delivers code that works. The harness is that environment for the model.
How this environment works: a loop
In practice, the environment runs in a loop: the model chooses the next action, the harness executes it, shows the result to the model, and decides whether to continue or stop.
You ask for a task
│
▼
┌──────────────── HARNESS ────────────────┐
│ │
│ assembles context ──► MODEL decides │
│ ▲ │ │
│ │ ▼ │
│ stores the result ◄── runs a │
│ and checks tool │
│ │ │
│ done? ──no──► repeat │
└──────────┼──────────────────────────────┘
▼ yes
Final answerA real example is Claude Code, which runs exactly this loop behind every task. It’s worth separating the harness, which is what runs, from frameworks and SDKs such as the Claude Agent SDK, which are used to build one.
The five main parts
Each turn of this loop depends on five parts. Think back to our newly hired developer: to work, they need a few basic things, and if one is missing, the work stalls. The same goes for the model. They are:
- The loop: decides when to call the model again, when to stop, and how many attempts to allow. It’s what keeps the agent from giving up too early or spinning on the same problem.
- The tools: define what the model can use, such as reading files, running commands, and searching the web. Without them, it can only describe what it would do.
- Context and memory: control what the model sees at each step and what gets noted between sessions. This is where yesterday’s work survives.
- Verification: tests and checks that tell whether the work is correct. Without it, the agent thinks it’s done and delivers something broken.
- Safety rules: define what the agent can do without asking permission, so that a wrong command doesn’t cause damage.
Why it matters: the same model, different results
If you’ve used the same model in two tools, you may have noticed: in Codex it behaves one way, in OpenCode another. I always wondered why, and the answer is in the harness.
Get Genildo cerqueira’s stories in your inbox
Join Medium for free to get updates from this writer.
In a 2026 study, a group of researchers, with Yunbei Zhang as first author, tested three models in three different harnesses, with 100 programming tasks. When they changed only the harness, results shifted between 8.5 and 13 percentage points. When they changed only the model, the shift was between 2.5 and 5 points. In this test, the environment mattered more than the model.
The same study gives a concrete case: Claude Opus 4.5 scored 45.9% in one harness and 55.4% in Claude Code, without changing the model. And a test by Endor Labs, a software security company, reported by MindStudio, found GPT-5.5 at 61.5% in Codex and 87.2% in Cursor.
The authors themselves warn that the experiment is small and covers only programming, so the numbers don’t hold for every case. But the direction is clear: when a model seems better in one tool than another, much of the difference may come from the harness.
This changes how you read any comparison between models. A benchmark score measures the model and the harness together, so the first question is which harness the test ran in. And when choosing a model, it’s worth testing it in the tool you will actually use. This doesn’t mean the model stops mattering: a good harness doesn’t replace a better model, and both count.
Two harness patterns Anthropic tested
Anthropic’s engineering team published two experiments showing, in practice, how a well-designed harness solves failures that no model solves on its own.
The first, described by Justin Young in November 2025, tackles a memory problem. Each agent session starts without remembering the previous one, as if our developer arrived every day without knowing what they did yesterday. This caused two failures: the agent tried to do everything at once and ran out of context midway, leaving the code half-finished; or it saw that progress already existed and declared the project done too early.
The solution split the work between two agents. An initializer agent prepares the ground: a script to bring up the environment, a progress file, a first commit, and a list of all features, each marked as not yet done. Then, in each session, a coding agent picks one feature, implements it, tests it, commits, and updates the progress. It can only mark a feature as done after testing it. This way, the list defines what “done” means, not the agent’s impression.
The second, described by Prithvi Rajasekaran in March 2026, tackles a judgment problem: a model tends to praise its own work, even when it’s weak. The solution was to separate who builds from who checks. One agent plans, another implements, and a third evaluates, navigating the application the way a user would. According to the post, it’s much easier to make a separate evaluator strict than to make the generator critique itself.
The two cases leave two lessons. When an agent fails, look at the system before rewriting the prompt: a tool, a verification step, or memory may be missing. And each part of the harness compensates for something the model still doesn’t do well on its own; when a better model arrives, it’s worth reassessing what is still needed.
Finally, not every task needs all of this. A simple, single-turn question doesn’t need a harness. It becomes worth it when the work has several steps, uses tools, and needs to be checked.
Sources
- Zhang et al., “Stop Comparing LLM Agents Without Disclosing the Harness”, arXiv 2605.23950
- Anthropic: Effective harnesses for long-running agents
- Anthropic: Harness design for long-running application development
- Claude Agent SDK documentation: how the agent loop works
- MindStudio, on the Endor Labs GPT-5.5 test
