Same Model, Different Harness, Different Score
NVIDIA’s own evaluation notes show an 8-point swing from the agent harness alone. A new 52,168-run study shows harness rankings flip by model and by task. Here’s what that means for leaderboards, and a 10-task test to run before you switch.
TL;DR
• An agent benchmark score belongs to a model and a harness together. NVIDIA ran its own Nemotron 3.5 Super through six harnesses: 49.45 on SWE-bench Pro in Claude Code, 41.04 in OpenCode. Same weights.
• A new study ran 4 harnesses with 4 models on 6 benchmarks. No harness won everywhere. Swapping the harness even reversed which of two models looked better.
• A clean exit isn't a solved task. One model finished every ClawMark run cleanly in Codex CLI and scored 11.96 out of 100. In OpenCode it finished 88% cleanly and scored 70.36.
• The harness also sets the bill. Token use differed by more than 10x between harnesses on the same model and benchmark, and a second study found up to 40x more tokens per solved task with pass rates within 8 points.
• Before you switch model or harness, run 10 of your own tasks in both harnesses, a few times each, and log success, clean exit, tokens and turns as separate columns.
When a new model comes out, the first thing most of us look at is a number. 41 on SWE-bench Pro, 60 on Terminal-Bench. We compare it with the number of the model we use now and decide whether it's worth switching.
That number is missing a column. An agentic benchmark doesn't run a model. It runs a model inside a harness: the agent program that writes the system prompt, exposes the tools, decides when to stop, retries failed calls and hands the result to the grader. Claude Code, Codex CLI, OpenCode, Pi and the rest are all harnesses. Change the harness and you've changed the thing being measured.
That's easy to say and easy to forget. Two recent sources make it concrete.
The evaluation page that came with a warning
NVIDIA's evaluation page for Nemotron 3.5 Super has a section most model docs don't. They ran the same model through six agent harnesses, with each harness's system prompt left unmodified, on two benchmarks:
Three things stand out.
The spread is large. On SWE-bench Pro, the best and worst harness are 8.41 points apart. On Terminal-Bench 2.1 it's 13.49 points, from Pi's 48.31 to Codex's 61.80.
The order changes with the benchmark. Claude Code is first on SWE-bench Pro and third on Terminal-Bench 2.1, where Codex leads. OpenCode is last on SWE-bench Pro but ahead of Mini-SWE-agent and Pi on Terminal-Bench.
NVIDIA tells you what to do with it. Their own sentence: "The harness ordering differs between the two benchmarks, so report the harness and its configuration with any agentic result." The same page also warns that "a gap against someone else's endpoint can come from their serving stack."
One detail matters if you compare these numbers with the rest of NVIDIA's results. For this table, NVIDIA removed Git history from the SWE-bench Pro repos and added an anti-cheating instruction. So the 41.04 in OpenCode here isn't the same setup as the 51.78 SWE-bench Pro score (also OpenCode) in the main results table on the same page. Same model, same harness, different benchmark setup, 10 points apart. That's the same lesson again, one level down.
It's on the leaderboards too
This isn't one vendor's quirk. Look at the leaderboards and you'll see model and harness listed as a pair.
The Terminal-Bench 2.1 board (now archived in favour of 4.0) ranks agent–model pairs, and its FAQ puts it plainly: "The same model can therefore receive different Terminal-Bench 2.1 scores when paired with different agent harnesses." Its release notes list 14 representative pairs: seven models, each run in its vendor's CLI and in Terminus 2, the benchmark's own reference harness.
GPT-5.4 mini scores 66.1% in Codex CLI and 36.9% in Terminus 2. That's 29.2 points, on the same 89 tasks. For GPT-5.4 it's 22.5 points. The vendor harness usually wins, which makes sense: it's tuned for its model. But not always. Gemini 3.1 Pro did better in Terminus 2 (70.7%) than in Gemini CLI (67.1%).
SWE-bench Pro shows the same thing in a footnote. Scale's public leaderboard runs models in SWE-Agent with a 250-turn limit by default. The top rows carry an asterisk: "Run with mini-swe-agent harness". So the first places on that board and the rest weren't measured with the same harness.
Vendors make the point when it suits them. Augment ran Claude Opus 4.5 on all 731 SWE-bench Pro public tasks in three agents: its own Auggie, Cursor and Claude Code. Auggie solved 15 more tasks than Cursor and 17 more than Claude Code. Scale's board has the same model at 45.89% in SWE-Agent, and Augment's 51.80% is "nearly 6 points" higher. Those are vendor-reported numbers about the vendor's own product, so treat them as a claim. But the shape matches everything else: same weights, different agent, different score.
The study: 52,168 runs, four harnesses
Single tables like NVIDIA's show that harnesses matter. They don't show how. That's what When Agents Fail: Model or Harness—Who's to Blame? (Zhang and others, Shanghai Artificial Intelligence Laboratory, 23 Oct) set out to measure.
The setup:
- Four general-purpose harnesses: OpenCode, Codex CLI, Pi CLI and DeepAgents (LangChain's).
- Four core models: DeepSeek V4 Flash Preview, MiniMax M3, Kimi K2.7 and Qwen3.6-35B-A3B.
- Two fixed native systems for reference: GPT-5.5 (xhigh) in Codex CLI, and DeepSeek V4 Pro in DeepSeek's own harness.
- Six benchmarks: SWE-bench-45 (the 45 SWE-bench Verified tasks that an experienced developer is expected to need at least an hour for), Terminal-Bench 2, VitaBench, LOCA-Bench, ClawMark and AgentSkillOS.
- Four repetitions of every run. 111 evaluation settings and 52,168 trajectories in total.
The paper's key move is to score two things separately. Task score is whether the agent got the answer right. Valid Completion Rate (VCR) is whether the run finished normally and handed the grader something it could evaluate, regardless of correctness. A harness can be good at one and bad at the other. They also compute a Fit score: how close a harness gets to the best joint result observed for a given model, on a 0 to 100 scale.
No harness wins everywhere
Averaged across models, there is a winner. OpenCode has the highest Cross-Model Fit, 87.48, ahead of Pi CLI (77.13), Codex CLI (73.62) and DeepAgents (64.20), on the six-benchmark panel with MiniMax, Kimi and Qwen. It stays first when you leave out any one model or any one benchmark. The places below it shuffle.
But "best on average" isn't "best for your model". For Qwen3.6, Codex CLI had the higher Fit (91.21 against OpenCode's 88.55). On LOCA-Bench, a long-context planning benchmark, Codex CLI was the best harness for every model. With MiniMax M3 it scored 87.00 against OpenCode's 21.00 (Mean-of-4), 63 points ahead of the runner-up. That MiniMax–Codex pairing even beat the GPT-5.5–Codex reference system (76.00) by 11 points. On that benchmark, the spread between the harness averages was 2.6 times the spread between the model averages.
The result that matters most if you pick models from leaderboards is this one: changing the harness can reverse the model ranking.
On SWE-bench-45, DeepSeek V4 Flash beats MiniMax M3 in Codex CLI, 36.11 to 30.56. In OpenCode, MiniMax beats DeepSeek, 62.78 to 32.22. If you'd read only one of those columns, you'd have picked a different model. Kimi K2.7 moves even more: 10.56 in Codex CLI, 48.89 in OpenCode.
None of the four harnesses is "bad". Each one fits some models and some tasks better than others. The authors' summary: harness rankings vary across models and benchmarks, and "similar task scores can accompany distinct execution profiles."
A clean exit isn't a solved task
The second finding is the one I'd want every agent dashboard to show.
The authors audited Kimi K2.7 on ClawMark, a benchmark of multi-day workflows where the agent has to actually change state, not just describe it. Two harnesses looked nearly perfect operationally and scored barely above 10 out of 100.
Codex CLI finished 100% of the runs cleanly and scored 11.96. Pi CLI finished 99.25% cleanly and scored 11.93. OpenCode finished only 88% cleanly but scored 70.36, and DeepAgents scored 58.65 with barely half its runs clean.
Get Andre Burgstahler’s stories in your inbox
Join Medium for free to get updates from this writer.
What went wrong in the clean runs? The authors looked for "action-realization gaps": the model's reasoning announces an action, and the response contains neither a tool call nor any visible text. They found that in 40.50% of Pi CLI attempts and 6.75% of Codex CLI attempts, and in none for OpenCode or DeepAgents. The run ends politely, the grader gets a valid handoff, and the work was never done.
It cuts the other way too. On SWE-bench-45, 25 runs solved the issue but weren't counted as valid completions, because they ended abnormally. And the native DeepSeek V4 Pro system finished every SWE attempt cleanly, yet 53 of its attempts still didn't resolve the issue. Even the GPT-5.5 reference in Codex CLI left more than a quarter of its Terminal-Bench 2 attempts non-clean.
So "it exited with status 0" and "it did the task" are two separate columns. If your own logs or CI only record the first, you're measuring the harness's manners, not its work.
A separate study of 255 bug reports from Codex, Gemini CLI, LangChain and CrewAI (Chen and others) gives this a name: "agent-reactive" bugs, which only show up when a particular model response triggers an abnormal reaction from the harness. They found that many of them are silent errors without a clear test oracle. That's exactly the kind of failure a model-only score can't show.
The harness also sets the bill
Pass rate is only half of the decision. The other half is what each success costs, and here the harness can matter even more.
In the Shanghai study, token use was heavily skewed. In 89 of 90 cells, the mean exceeded the median, and within the same model and benchmark, one harness could use more than ten times the tokens of another. Which harness was lightest changed from model to model. The tail was the expensive part: the top 5% of Qwen–Pi runs on SWE-bench-45 used about 83% of that configuration's tokens, and failed attempts accounted for almost all of it.
On average, Codex CLI (0.877) and Pi CLI (0.906) ran lighter than DeepAgents (1.095) and OpenCode (1.150), on the paper's relative-load scale. But OpenCode had the highest share of pairings on the cost–quality frontier (61.11%), because it often bought its extra tokens with better results.
The Scaffold Effect in Coding Agents (Vats and Golev) is a useful counterweight. They ran Qwen 3.6 Plus and MiniMax M2.5 in Goose, OpenCode and the OpenHands SDK on 50 Terminal-Bench Pro tasks. The paired pass-rate differences between harnesses were small, 0 to 8 points, and only the largest gap had a confidence interval that excluded zero. But tokens per solved task differed by up to 40x. The failure patterns also repeated across both models, so each harness had its own signature: reasoning failures in Goose, verification and turn-limit failures in the OpenHands SDK, idle loops and timeouts in OpenCode.
So two harnesses can look almost tied on a leaderboard and be very different on your bill.
One more number from the Shanghai study is worth keeping in mind when you read anyone's results, including your own. Best-of-4 was higher than Mean-of-4 in all 76 configurations they compared. For MiniMax in OpenCode on SWE-bench-45 the gap was 19.4 points. A single lucky run tells you what the pairing can do, not what it usually does.
So what does a leaderboard number mean?
It means “this model, in this harness, with this configuration, on these tasks”. Good leaderboards are moving towards saying that out loud.
Terminal-Bench lists the agent next to the model in every row. Artificial Analysis runs its agentic evaluations in harnesses it controls: Terminal-Bench 2.1 in Terminus 2, and FrontierCode 2.0 and CritPt in its own harness, Stirrup. Its new Intelligence Index v5 (27 Oct) also puts 60% of the weight on evaluations with private test sets or held-out answers, across 9 evaluations. That helps against contamination. It doesn’t change the basic point: the score was measured in their harness, not yours. Harness-Bench (106 tasks, 5,194 trajectories) goes further: its authors conclude that agent capability should be reported at the model–harness configuration level rather than attributed to the base model alone.
And the harness isn’t fixed even for one tool. Meta-Harness searches automatically for better harness code, and its discovered harnesses beat the best hand-engineered baselines on TerminalBench-2. Your Claude Code with your hooks, skills and AGENTS.md is already a different harness from the one in the vendor’s chart.
That doesn’t make leaderboards useless. Use them to build a shortlist. A model that’s 20 points behind in every harness is unlikely to win in yours. But when two candidates are within 10 points, or the score comes from a harness you don’t use, the leaderboard can’t decide for you. Your own tasks can.
A 10-task, two-harness test
The Shanghai authors end with advice for harness builders: “Adapt resource limits to the model and task, bound failed retries, and detect stagnation and repeated actions. Reserve verification and handoff budgets.” For the rest of us, the practical version is a small eval before any switch. Here’s the one I’d run. It takes an afternoon, mostly unattended.
1. Pick 10 tasks from your own recent work. Real tickets or commits from the last month, each with a check you can run: a test, a script, a build. Mix them: a few bug fixes, a few small features, a refactor, one long task that usually takes you an hour. Freeze the starting commit for each.
2. Pick two pairings and change one thing. Either the same model in two harnesses (your current one and the candidate), or two models in the harness you actually use. Don’t change both at once. Keep everything else equal: model version, reasoning effort, AGENTS.md, permissions, MCP servers, time limit and turn limit.
3. Run each task three times per pairing. That’s 60 runs. One run per task is a lottery, as the Best-of-4 gap shows. Three gives you a rough mean.
4. Log the columns separately. One row per run:
task,pairing,run,check_passed,clean_exit,handoff_ok,turns,tool_calls,input_tokens,cache_read_tokens,output_tokens,cost_usd,wall_min,notes
# illustrative row, not measured data:
T03,modelA+harnessX,2,no,yes,yes,41,63,812000,640000,18400,1.92,14,"said it would run tests, never did"check_passed is your test. clean_exit is whether the harness finished normally (no crash, timeout or stuck loop). handoff_ok is whether it left something you could check, like a diff or a report. Keep cache reads separate from fresh input, because they're priced very differently (I wrote about why in Trimming Context Can Make Your Agent More Expensive). If you use Claude Code, Codex or Gemini CLI, their OpenTelemetry export gives you these counts per run, though each counts cached input differently; I covered how to line them up in Claude Code, Codex and Gemini CLI Just Started Speaking the Same Cost Language.
5. Read four numbers per pairing. — Mean success: tasks passed, averaged over the three runs. Not “solved at least once”. — Clean but wrong: runs with clean_exit=yes and check_passed=no. These are the dangerous ones, because nothing looks broken. - Solved but messy: runs that passed the check but crashed or timed out. These are fixable with harness settings. - Median and worst cost per success. The tail is where the money goes.
6. Decide with a margin. With 10 tasks, one task is 10 points, and that’s noise. My rule of thumb, not a statistical test: switch only if the candidate solves at least two more tasks on average without a worse cost tail, or matches your current pairing at a clearly lower median cost. If it’s closer than that, stay put and rerun when the next version ships.
Keep the task set. The next time a model or harness update lands, it’s a one-command rerun instead of a debate.
What the papers don’t show
- Not your tasks. The Shanghai study used four open-weight models and six public benchmarks. No Claude models, and GPT-5.5 only as a fixed reference. Claude Code wasn’t one of its four harnesses.
- Versions move fast. The paper pins OpenCode v1.15.6 and Pi CLI 0.79.3; the harnesses you install today are newer. A ranking from October can be stale by December.
- Terminal-Bench 2 isn’t 2.1. The paper used Terminal-Bench 2; NVIDIA and the leaderboard figures use 2.1, which fixed 28 of the 89 tasks. Don’t compare scores across versions.
- NVIDIA’s table is one model. It shows the spread for Nemotron 3.5 Super, not a general harness ranking.
- Vendor numbers are vendor numbers. Augment’s comparison is about its own product.
- All the papers cited here are preprints, not yet peer reviewed.
- Fit and VCR are the authors’ metrics. They’re useful, but your
check_passedcolumn is the one that counts.
Sources
- Shao Zhang, Hangfan Zhang, Yuanyuan Shi and others (Shanghai Artificial Intelligence Laboratory), “When Agents Fail: Model or Harness — Who’s to Blame?”, arXiv 2610.32483, 23 Oct 2026 (read 3 Nov 2026; preprint)
- Vats and Golev, “The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation”, arXiv 2607.22585 (read 3 Nov 2026; preprint)
- Yao and others, “Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows”, arXiv 2605.27922 (read 3 Nov 2026; preprint)
- Lee and others, “Meta-Harness: End-to-End Optimization of Model Harnesses”, arXiv 2603.28052 (read 3 Nov 2026; preprint)
- Chen and others, “Understanding Agent-Reactive Bugs at the Model-Harness Boundary”, arXiv 2607.15684 (read 3 Nov 2026; preprint)
- NVIDIA, Nemotron 3.5 Super evaluation and model card (read 3 Nov 2026)
- Terminal-Bench, Terminal-Bench 2.1 release notes and the 2.1 leaderboard (read 3 Nov 2026)
- Scale, SWE-Bench Pro public leaderboard (read 3 Nov 2026)
- Augment, Auggie tops SWE-Bench Pro (vendor post, read 3 Nov 2026)
- Artificial Analysis, Announcing Intelligence Index v5, Intelligence benchmarking methodology and Terminal-Bench v2.1 (read 3 Nov 2026)
- Previous articles: Trimming Context Can Make Your Agent More Expensive, Claude Code, Codex and Gemini CLI Just Started Speaking the Same Cost Language
