Evaluating Coding Agents
Evaluating coding agents: green tests are necessary, not sufficient — grade the diff a reviewer would merge.
Every coding-agent number you have seen answers the question “did the hidden tests pass?” The question you actually have is “would my team merge this diff, how often, and at what price?” Those are different questions, and the gap between them has now been measured. This playbook builds the eval that answers the second one: tasks mined from your own merged pull requests, a containerised harness with held-out tests, a grading layer above the tests, repeated runs that measure reliability rather than luck, and cost and wall-clock per task — wired into CI so that every model, prompt or tool change gets scored before it ships.
Step 1Why the public number cannot answer your question.
Treat this step as motivation, not literacy; for how to read any leaderboard row see reading benchmarks. The point here is narrower: the best-known coding benchmark has been audited in both directions, and it failed both.
- It was carefully built. SWE-bench grades an agent by applying its patch to a real repository and running the issue’s fail-to-pass tests plus the existing pass-to-pass tests. SWE-bench Verified (OpenAI, August 2024) came from 93 Python developers screening 1,699 random SWE-bench samples for underspecified issues and unfair tests, keeping 500.
- Its tests accept wrong patches. “Are ‘Solved Issues’ in SWE-bench Really Solved Correctly?” (ICSE 2026) re-ran agents’ plausible patches against the full developer test suites and differential tests: 29.6% behave differently from the gold patch, 28.6% of those divergent patches are certainly incorrect on manual inspection, and together the weaknesses inflate reported resolve rates by 6.2 percentage points. UTBoost (2025) generated extra tests, found 169 Verified patches wrongly marked as passing, and changed 24.4% of leaderboard rankings.
- Its tests also reject right patches. On 23 February 2026 OpenAI stopped reporting Verified. Engineers audited 138 problems that o3 failed consistently across 64 runs and found at least 59.4% had flawed tests or descriptions that reject functionally correct submissions; the same post cites contamination, with frontier models able to reproduce gold fixes from memory. It recommends SWE-bench Pro (Scale AI, September 2025: 1,865 tasks from 41 repositories, with held-out and proprietary commercial subsets kept private). Terminal-Bench 2.0 (89 human-checked containerised tasks, graded on final container state) is the other hard successor.
- Your own perception is not an instrument either. METR’s randomised trial (July 2025) had 16 experienced open-source developers work 246 real issues in their own repositories: with AI tools allowed they took 19% longer, after forecasting a 24% speed-up and still believing afterwards they had been sped up by 20%. METR’s February 2026 follow-up says newer tools likely help more but calls its own late-2025 data unreliable because of selection effects. The lesson is not the sign of the result; it is that the people using the tool could not tell.
Both failure directions share one cause: a test suite is a partial specification, written by someone who was not thinking about the agent’s patch. A green run is evidence the patch is not obviously wrong. Whether it is mergeable is a separate measurement, and nobody publishes it for your repository.
Step 2Mine tasks from your own merged pull requests.
Your merge history is a free, labelled dataset: every merged PR is a task your team considered done, with a diff a reviewer accepted. Reconstruct SWE-bench’s recipe on it.
- Select PRs that touch both source and tests. Split each diff into the fix (non-test files) and the test patch. Check out the parent commit, apply only the test patch, and run it: the tests that fail there and pass after the fix are your fail-to-pass set; everything else that passes in both is pass-to-pass. A PR with no fail-to-pass test is not gradeable by execution — keep it only for the rubric track in step 4.
- Write the prompt from the linked issue, not the PR. PR descriptions leak the solution (“switch to a reentrant lock in
Pool.acquire”). Use the issue or ticket text as filed, and strip any line that names the fix. - Stratify, then stop at a hundred or so. Bucket by kind (bug fix, feature, refactor, dependency, config) and by gold-diff size, and sample to match the work you actually intend to delegate. Fifty to 150 tasks is enough to see regressions of a few points with repeat runs; more tasks you cannot afford to rerun weekly are worth less than fewer you can.
- Prefer PRs merged after the model’s training cutoff. If your repository is public, older fixes may be memorised exactly as Verified’s were. Record the merge date on every task so you can report scores split by pre- and post-cutoff.
# tasks/pay-1482.yaml — one task mined from a merged PR id: pay-1482 source_pr: "#1482" # merged 2026-08-19, after the model cutoff base_commit: 9f3c2e1 # parent of the merge; fix reverted, tests absent kind: bugfix prompt: issues/1467.md # the issue as filed, solution hints removed hidden_tests: # copied in only after the agent stops fail_to_pass: [tests/test_refunds.py::test_partial_refund_idempotent] pass_to_pass: tests/ # full suite must stay green gold: { files: 2, lines_changed: 31 } budget: { wall_clock_min: 20, usd: 3.00 }
Step 3Build the harness: one container per task, tests the agent never sees.
The harness is the instrument. Make it boring and reproducible, because every uncontrolled variable in it shows up as noise you will later attribute to the model.
- Pin the environment per task. Build an image at
base_commitwith dependencies installed and cached, network off during the run, and the same toolchain CI uses. The agent works inside it with exactly the tools it will have in production — see sandboxing and execution. - Hide the tests. The fail-to-pass tests are not in the image while the agent works. When it stops, the harness extracts its diff, applies it to a fresh container, copies the hidden tests in, and runs them. An agent that can read the grading tests is being graded on reading.
- Validate every task twice before trusting it. The gold fix must pass and the bare base commit must fail; run the gold three times to catch flaky tests. Tasks that fail validation are bugs in your eval, not hard problems — exactly the class OpenAI’s audit found in Verified.
- Freeze the scaffold. System prompt, tool set, retry policy, context strategy and budgets are part of the configuration under test and are versioned with the results. Change two at once and the comparison is meaningless.
- Keep the trajectory. Store every tool call and its output. When a score moves, the trajectory is the only place the reason lives.
Step 4Grade beyond the tests: would a reviewer merge this diff?
This is the step the public benchmarks skip and the reason to build your own. Hidden tests become a gate; mergeability is the score.
- Hard fails before anything else. The diff edits, skips or deletes an existing test; adds a broad
exceptor a sleep to get green; touches files outside the task’s plausible scope; or breaks the linter or type checker. Each is an automatic zero regardless of test results — the agent version of weakening the assertion. - Differential checks against the gold patch. Run any extra tests you have (property tests, the module’s wider suite, a fuzz pass) on both the agent patch and the gold. Behaviour that diverges from the merged fix is the 29.6% bucket from step 1: not necessarily wrong, but it needs a human look.
- Diff-size ratio. Lines changed divided by the gold’s. A ratio far above one usually means drive-by refactors, duplicated helpers or defensive clutter that reviewers reject; far below often means a special case that only satisfies the test.
- A rubric with three verdicts. Merge as-is, merge after edits, reject — scored against written criteria: follows repository conventions, reuses existing abstractions, includes a test of its own, no unrelated changes. Start with engineers grading a sample; move to an LLM judge only once it agrees with them at a rate you have measured, and keep re-checking it, per judge calibration. The same precision-first discipline as code review agents applies.
# one task, two configurations, three runs each task config hidden_tests hard_fail diff_ratio verdict pay-1482 A pass - 1.1 merge as-is pay-1482 A pass - 0.9 merge as-is pay-1482 A fail - - reject pay-1482 B pass test edited 2.4 reject pay-1482 B pass - 3.8 merge after edits pay-1482 B pass - 1.0 merge as-is # tests-only score: A 2/3, B 3/3 → B looks better # mergeable score: A 2/3, B 1/3 as-is (2/3 incl. edits) → A is better
The table is the whole argument in miniature. A configuration that learned to get tests green by any means will beat one that writes reviewable code on the metric every leaderboard reports, and lose on the one your team experiences.
Step 5Run every task several times, and report reliability, not best-of.
Agent runs are non-deterministic; one run per task measures a draw. Run each task n times (three to five is the practical floor) and report two different numbers, because they answer different questions.
# per task: n runs, c of them mergeable. Average each over tasks. pass_at_k(n, c, k) = 1 - C(n-c, k) / C(n, k) # any of k succeeds: headroom pass_caret_k(n, c, k) = C(c, k) / C(n, k) # all k succeed: reliability # n=5, c=3: pass@1 = 0.60 pass@3 = 1.00 pass^3 = 0.10
- pass@1 is the expected rate of a single unattended run, which is what a background agent gets.
- pass^k is what trust feels like. Introduced by τ-bench (2024), where gpt-4o succeeded on under half of retail tasks and its pass^8 fell below 25%, it asks whether the agent solves the task every time. A task with pass@1 of 0.6 is one your team will stop delegating after the second bad week.
- pass@k is not a deployment number. It assumes something picks the winning attempt, and in production that something is a human reading k diffs.
- Compare configurations paired, on the same tasks. Report per-task wins and losses, not only two averages, and treat a few points of movement on a hundred tasks as noise until repeat runs say otherwise. Watch for tasks that flip between runs; they are where the next regression will appear first.
Step 6Price it per mergeable diff, then put it in CI.
A configuration that is five points better and three times the cost may be the wrong choice; you cannot know without measuring both on the same runs.
- Record tokens, dollars and wall-clock per run, including failed runs. Then divide total spend by the number of mergeable diffs, not attempts and not test passes. Cost per merge-as-is diff is the number that belongs next to an engineer’s hourly rate, and it is the number METR’s result says you will not estimate correctly by feel.
- Track the tails. Median and 90th-percentile wall-clock per task; a configuration whose p90 run takes forty minutes changes how people use it even if the median is fine. Note the human minutes spent on merge after edits verdicts — that is cost too.
- Gate changes in CI. Any change to model, system prompt, tool definitions or scaffold triggers the suite; the pull request gets a comment with pass@1, pass^k, hard-fail rate, cost per mergeable diff and p90 wall-clock against the current baseline, and blocks on regression past thresholds you set in advance. The mechanics are in eval-driven development and CI.
- Refresh the set, retire tasks. Mine new tasks from each month’s merges, drop ones every configuration solves every time, and keep a post-cutoff slice so contamination cannot creep back in through your own public history.
A leaderboard tells you which harness passed someone else’s partial tests on code that may have been memorised; your own eval tells you how often an agent produces a diff your reviewers would merge, how reliably, and what each one costs. Build the second, from your merged PRs, with tests the agent never sees and a reviewer’s verdict on top — it is the only coding-agent number that predicts what happens on Monday.