The top 10 agent harnesses tells you which coding systems exist and how they differ. The next buyer question is harder: when those agents enter a real repository, which ones produce a correct change rather than a confident diff?
Public coding evals offer evidence, but the headline score often collapses model, harness, repository, tests, effort, and cost. This guide reads 2026 results as system measurements and gives teams a private-repo protocol that survives the next model launch.
TL;DR: public scores are a shortlist, not a winner
| Question | Direct answer |
|---|---|
| Best public repo-repair signal? | SWE-bench Pro, with broken-task and contamination caveats |
| Best terminal-work signal? | Terminal-Bench, which tests environment interaction |
| Do rankings agree? | No. GPT-5.5 and Claude swap leadership across published tables |
| What is being scored? | Model + harness + tools + environment + budget |
| What should a team measure? | Accepted change rate, repair time, regression risk, cost, and latency |
| How should you choose? | Run 20–50 private historical tasks under equal budgets |
The current public snapshot
OpenAI's GPT-5.5 launch table reports the following public signals:
| System/model label | SWE-bench Pro (Public) | Terminal-Bench 2.0 |
|---|---|---|
| Claude Opus 4.7 | 64.3% | 69.4% |
| GPT-5.5 | 58.6% | 82.7% |
| GPT-5.4 | 57.7% | 75.1% |
| Gemini 3.1 Pro | 54.2% | 68.5% |
This table does not support “GPT-5.5 is the best coding agent” or “Claude is the best coding agent.” It supports two narrower statements under OpenAI's listed setups: Claude scored higher on the public long-horizon repository repair set, and GPT-5.5 scored higher on Terminal-Bench 2.0.
OpenAI footnotes memorization evidence on SWE-bench Pro and later estimated roughly 30% of its tasks are broken. Our benchmark fact-check covers that audit. A damaged benchmark can still preserve broad signal while making close differences unreliable.
What the benchmarks actually test
SWE-bench Pro: issue-to-patch repository work
SWE-bench tasks begin with real repository issues and tests. An agent must inspect a codebase, infer the intended change, edit files, and satisfy evaluation. Pro uses more complex, long-horizon tasks and includes public, held-out, and commercial repository partitions designed to reduce contamination.
It is closer to engineering than a function-completion benchmark, but still narrower than a job. The agent does not attend product meetings, negotiate requirements, own an incident, or maintain the feature six months later.
Terminal-Bench: working inside an environment
Terminal-Bench evaluates whether an agent can complete tasks through a terminal: inspect files, run commands, install or configure components, and manage state. It exposes a different capability than repository patching. A system can be excellent at shell-driven execution and weaker at understanding a large codebase's design.
Senior SWE-bench and newer hard sets
Newer benchmarks target tasks closer to senior engineering: cross-file reasoning, architecture, difficult debugging, and longer trajectories. They fight saturation, but a harder name does not automatically guarantee clean tests. Inspect task construction, held-out policy, and evaluation code.
Your repository: the benchmark that decides procurement
Public sets rarely match your languages, build times, monorepo boundaries, test coverage, internal frameworks, security policy, or definition of a reviewable patch. A model can dominate Python bug repair and struggle with your TypeScript application plus infrastructure configuration.
Why the harness changes the score
An agent harness decides:
- Which repository files enter initial context.
- Whether the agent can search, run tests, browse docs, or use an LSP.
- How command timeouts and permissions work.
- Whether failed tests trigger automatic retries.
- How context is compacted over a long run.
- Whether subagents can investigate in parallel.
- How the final patch is extracted and submitted.
The model reasons within those choices. A strong model in a weak harness can fail to find the relevant file. A good harness can give a smaller model better retrieval, verification, and recovery.
This is the natural sequel to the harness ranking: compare systems under the workflow you will deploy, not base models in a vacuum.
Five ways coding-agent comparisons become unfair
Different effort budgets
One system receives high reasoning, five retries, and 30 minutes; another receives default effort, one attempt, and ten minutes. Accuracy is not comparable until budget travels with the score.
Native versus generic scaffolds
Claude may be deeply tuned for Claude Code, GPT for Codex, and Gemini for Antigravity. A native-harness comparison measures the product users buy, which is useful. A unified harness isolates model behavior. Label which question you answer.
Different test visibility
An agent with access to evaluation tests can optimize toward them. Hidden tests better measure generalization, but environment problems can reject valid solutions. Keep visible developer tests separate from hidden acceptance tests.
Contaminated repositories
Popular public issues and patches may appear in training data. Held-out or newly created tasks reduce this risk. OpenAI retired SWE-bench Verified as a frontier measure partly because memory increasingly polluted the signal.
Excluding expensive failures
Environment setup failure, refusal, timeout, and exhausted context are product failures. Report them separately for diagnosis, but include them in the deployed success rate.
Build a private repository eval
Step 1: select historical tasks
Choose 20–50 resolved issues that were not trivial and whose expected patches remain private to the evaluation runner. Include:
| Task family | Suggested share |
|---|---|
| Bug fixes with regression test | 30% |
| Small feature additions | 20% |
| Refactors preserving behavior | 15% |
| Test creation or repair | 15% |
| Configuration/build issues | 10% |
| Documentation tied to code truth | 10% |
Remove tasks that depend on unavailable private services unless the harness can provide a deterministic substitute.
Step 2: define success before running
Require build/test pass, hidden regression tests, lint/type checks where relevant, no prohibited file changes, and human review. Add task-specific assertions. “Patch applies” is not engineering success.
Step 3: freeze the environment
Use a container or reproducible VM, pin the repository commit and dependencies, record network access, and impose the same CPU/memory/time limits. Environment drift creates fake model differences.
Step 4: equalize budget
Pick a dollar or wall-clock budget per task. If models have different token prices, equal dollars answers a procurement question; equal tokens answers a different capability question. Report both where possible.
task_budget:
max_minutes: 30
max_attempts: 2
max_model_spend_usd: 8
network: allowlisted_docs_only
tests: visible_subset_plus_hidden_suite
Step 5: blind the review
Hide model identity from reviewers. Score correctness, scope discipline, maintainability, security, and review effort. Run automated tests first, but do not let a green suite excuse an obviously dangerous implementation.
The metrics that matter
| Metric | Why it matters |
|---|---|
| Accepted change rate | Final deployable outcome after review |
| Test pass rate | Objective behavior under visible/hidden checks |
| Human repair minutes | Cost transferred to engineers |
| Regression/severity rate | Tail risk hidden by average pass rate |
| Cost per accepted change | Procurement and routing value |
| P50/P95 completion time | Daily workflow predictability |
| Files changed outside scope | Agent discipline |
| Retry and timeout rate | Harness stability |
The numerator for cost should include failed attempts. If a $2 model succeeds half the time and a $5 model succeeds 90%, their cost per accepted fix can be similar before review time.
A worked selection example
Suppose 40 private tasks produce:
| Agent | Accepted | Model spend | Repair hours | Cost/accepted before labor |
|---|---|---|---|---|
| Agent A | 26 | $160 | 18 | $6.15 |
| Agent B | 30 | $280 | 8 | $9.33 |
| Agent C | 22 | $70 | 24 | $3.18 |
If reviewer time costs $80/hour, Agent B may be cheapest overall despite the highest API bill:
Agent A: $160 + (18 × $80) = $1,600 total
Agent B: $280 + (8 × $80) = $920 total
Agent C: $70 + (24 × $80) = $1,990 total
This is why token price tables do not pick coding agents. Review burden dominates.
What people are asking
Should we test the model or the product?
Test the product configuration you will deploy first. Add a unified-harness experiment when you are deciding whether to build your own orchestration or switch providers inside an open harness.
How often should the eval run?
Run a smaller regression suite on model/harness updates and the full suite before a major default change. Preserve snapshots because moving aliases can change behavior without a new procurement decision.
Can an agent write its own tests?
Yes, but self-written tests are evidence, not independent proof. Use hidden tests or reviewer-authored checks to catch reward hacking. Our coverage of Cursor and SWE-bench reward hacking shows the failure pattern.
What if no agent clears the bar?
Route narrower tasks: test generation, documentation updates, dependency cleanup, or bounded bugs. An agent does not need to own whole features to create value.
Public benchmarks tell you where to investigate. A frozen, reviewed, costed private suite tells you what to deploy. The second result is less viral and far more valuable.
The minimum artifact for a credible coding-agent result
A reproducible result should ship more than a percentage. Publish the task manifest and commit hashes, environment image or lockfile, agent and model versions, tool permissions, prompt or configuration, time and token budgets, retry rules, test commands, and the final patch for every attempt. Identify tasks removed after inspection and explain why.
For each accepted patch, report whether tests alone approved it or a maintainer reviewed design quality. For failures, preserve the trace long enough to classify navigation, understanding, editing, testing, environment, and stopping errors. This turns the benchmark into engineering data instead of a launch graphic.
If confidentiality prevents public traces, an internal review packet can still contain the same fields. The point is accountability: another evaluator should be able to distinguish a better model from a better scaffold, a larger compute budget, or a forgiving judge. Without those artifacts, “real repos” describes the dataset's origin, not the realism of the evaluation.
Related on explainx.ai
- Top 10 open and closed-source agent harnesses
- AI benchmark claims fact-checked
- OpenAI's SWE-bench Pro audit
- Terminal-Bench 2.0 guide
- Agent harness engineering: seven planes
- How to build an enterprise AI benchmark
Public scores and audit status are accurate as of July 26, 2026. Model snapshots, harness commits, benchmark tasks, and provider defaults change; preserve the exact configuration behind every result.