Six months after shipping 89 tasks that 93 contributors had written and three expert reviewers had screened, Terminal-Bench's maintainers re-audited their own benchmark and had to repair between 26 and 28 of them. That is the most useful number across all four of these benchmarks, and it is not a score: it is a measured decay rate for a carefully built eval set, and it is the prior you should apply to your own.
At a glance
All four measure roughly the same thing — can an agent resolve a real software task in a real repository — and they differ almost entirely in what they do to stop their own items from rotting.
| Benchmark | Tasks | Its defence against decay | What that leaves open |
|---|---|---|---|
| SWE-bench Verified | 500 | Human solvability screening, once: 93 Python developers filtered 1,699 candidates. | Frozen since August 2024; leakage measured by outsiders, not maintainers. |
| SWE-bench Pro | 1,865 published (731 public, 858 held out, 276 commercial) | A private mirror set, plus copyleft repositories as a stated legal deterrent to training on them. | Deterrent is reasoning, not a measurement; no ongoing refresh. |
| Terminal-Bench 2.1 | 89 | Maintainers re-audited and repaired their own tasks six months in. | No contamination mechanism at all. |
| SWE-rebench | 21,336 mined; 294 in the paper's set; 860 curated | Continuous re-mining, monthly splits, and items flagged against model release dates. | Auto-mined tasks get no expert review; validity is inferred from execution. |
SWE-bench Verified
What the curation actually bought
Verified is the 500-problem subset released on 13 August 2024 with OpenAI's Preparedness team, built by having 93 Python developers screen 1,699 candidate instances for two specific defects: issue descriptions too vague to be solvable, and unit tests scoped so loosely that a wrong patch would pass. That is a validity audit, done once, by hand, and it is why Verified remains the most trusted of the four for the narrow question it answers.
The screening did not remove the other failure. An outside audit — the SWE-bench+ work — found that 37 of the 500 Verified instances, about 7%, contain the solution in the issue description or the GitHub discussion the task points at. That is the defensible per-task figure; larger numbers circulate from broader denominators, including a 55% share of Verified fixes classified as suspicious once weak tests are counted alongside leakage, pooled over a smaller set of commonly resolved issues. Cite the 7% if you cite one.
The leaderboard became a different object
Two changes matter more than any score on it. Since 18 November 2025 the Verified and Multilingual leaderboards accept submissions only from entrants with a research publication and an academic or research-lab affiliation, which makes the board a curated record rather than an open contest. And bash-only runs are a filter on the Verified board rather than a separate split, which is why aggregator sites quoting "SWE-bench Verified" disagree with each other by roughly twenty points: a minimal bash agent and an unconstrained commercial harness are not measuring the same quantity, and a number without its harness is not a number.
Running it yourself is not free either. The repository recommends an x86_64 machine with at least 120 GB of free storage, 16 GB of RAM and 8 CPU cores, with arm64 support still experimental. That figure is the honest floor for anyone who plans to stop reading leaderboards and reproduce one.
SWE-bench Pro
Contamination as a licensing strategy
Pro's distinguishing move is structural rather than procedural. As published it holds 1,865 problems across 41 repositories, split into 731 public, 858 held out and 276 commercial, where the held-out set mirrors the public one but draws from separate repositories and stays private specifically to monitor overfitting. The public and held-out repositories were chosen under strong copyleft licences, and Scale's stated reasoning is that the licence "acts as a legal deterrent against the code's inclusion in model training data."
Read that as the hypothesis it is. Nobody has published a measurement of whether GPL licensing changes what ends up in a pre-training corpus, and third-party write-ups that escalate it to contamination being structurally impossible are adding a claim Scale did not make. Scale is admirably blunt about the general problem on its own leaderboard: models "have likely seen the evaluation code during training, making it hard to know if they are problem-solving or recalling a memorized solution."
The public split has already moved
If you are quoting Pro's numbers, check which generation you have. The live public dataset is now a validated V2 of 642 tasks across 11 repositories, where V1 was the original 731, and there is a HARD-51 subset alongside it. The paper-era 1,865 / 731 / 858 / 276 figures are still correct about the paper and wrong about the download. The repository also carries its own notes about leaderboard problems being worked on and results being republished without a cap limit, which is the kind of maintenance log that should raise your confidence in a benchmark and lower your confidence in any single score you read off it.
Terminal-Bench 2.1, and Harbor underneath it
The only published self-audit of task validity
Terminal-Bench 2.0 shipped on 7 November 2025 alongside Harbor, the Apache-2.0 harness that runs it. Its construction was unusually selective: 93 contributors produced 229 candidate tasks and 89 were selected, a 39% acceptance rate, with the author's own difficulty rating plus review by three experienced humans.
Then, on 6 May 2026, the maintainers published 2.1 with a sentence no other benchmark on this list has published about itself: they were releasing it "to fix issues in 28 of the 89 tasks in Terminal-Bench 2.0." Nine of those had external dependencies that changed after the benchmark was built. Eight had insufficient resource budgets for at least one valid approach — that is, the task was solvable in principle and not within the limits the harness gave the agent. A third group, which the post does not count, had instructions that were not aligned to their tests. The 2.1 repository's own README says 26 tasks were modified rather than 28, so quote the range and the disagreement rather than picking a side.
Take the arithmetic seriously. Somewhere between 29% and 31% of a hand-built, expert-reviewed, 89-task benchmark had decayed or been misspecified within six months of release, and the maintainers found it only because they went looking. The claim that follows — "after these changes, no task is unsolved in Terminal-Bench 2.1" — is what a benchmark looks like when someone measures its ceiling instead of assuming it.
What Harbor changes about running any of them
Harbor is the reason this benchmark is easy to re-run and therefore easy to re-audit. It evaluates arbitrary agents, including Claude Code, OpenHands and Codex CLI; it lets you publish your own benchmarks and environments; and it parallelises a run across sandbox providers — Daytona, Modal, Blaxel and others — so a local Docker run and a thousand-environment cloud run are the same command with a different flag. SWE-bench and SWE-rebench are both listable through it. One practical trap: the run identifiers differ between versions, terminal-bench@2.0 against terminal-bench/terminal-bench-2-1, and scores across the two are not interchangeable, since repairing broken tasks moves most agents up.
SWE-rebench
Outrunning contamination instead of filtering it
SWE-rebench takes the one position the other three do not: if indirect leakage cannot be filtered, produce new items faster than they age. Its automated pipeline mined 21,336 verifiable tasks from 3,468 Python repositories; the paper evaluated a fixed set of 294 tasks from 169 repositories; the curated leaderboard pool distributed through Harbor Hub is 860 tasks with pre-built container images per instance, organised into monthly splits you can recover by filtering on the created_at field.
The contamination control is worth describing precisely, because secondary summaries get it wrong. It is not a rolling window that only ever uses post-release tasks; it is a flag. Evaluations that include issues created before a model's release date are marked as potentially contaminated on the board, and the reader decides. That is weaker than exclusion and considerably more honest, because it keeps the comparison possible while labelling it.
What you give up
Nothing in that pipeline screens for the defects Verified's 93 developers were hired to find. Validity is inferred from executability — the task builds, the tests run, the reference patch passes — which catches broken environments and catches nothing about a vague issue or a test too weak to discriminate. The paper's motivating finding, that open models score lower on freshly mined tasks than on SWE-bench Verified while relative rankings largely survive, is directionally the point of the whole project; the specific per-model comparison figures circulating in secondary coverage are not reliable enough to quote.
Cross-cutting: three axes that actually separate them
Who measures the ceiling
Only Terminal-Bench has published a number for how many of its own tasks were unsolvable. Verified's screening was a one-time filter applied before release, not a repeated measurement; Pro's public-split revision from 731 to 642 tasks is a revalidation but is not accompanied by a defect taxonomy; SWE-rebench's executability check is continuous but shallow. Across all four, exactly one maintainer team has told you what fraction of its denominator was dead, and the answer was about three in ten.
Who tells you the number's variance
SWE-agent's own appendix is the cleanest primary statement: aggregate performance variance is relatively low, while per-instance resolution "can change considerably" between runs. SWE-rebench publishes standard error alongside resolved rate and pass@5, which is the only maintainer-published variance figure among the four. Everyone serious in the third-party world now averages three to five repeats and prints an interval. If you are comparing two systems a point apart on any of these, you are reading noise — the arithmetic is in eval variance and statistical power.
Who you are allowed to be
This is the axis nobody puts in a comparison table and it decides what the boards mean. Verified now requires a publication and an institutional affiliation to submit. Pro runs separate public and private leaderboards and has acknowledged working through leaderboard problems. Terminal-Bench's scores arrive through a harness anyone can run. SWE-rebench is maintained by a cloud vendor whose own models appear on it. None of that is disqualifying, and all of it changes how you weight a row.
When to pick which
| If you need… | Use | Because |
|---|---|---|
| A comparison against published history | SWE-bench Verified, with the harness named | Four years of submissions, screened items, and a known ~7% leakage figure you can discount for. |
| Evidence a model has not memorised the set | SWE-rebench, newest monthly split | Items postdate the model, and the ones that do not are flagged rather than hidden. |
| Long-horizon, environment-heavy work | Terminal-Bench 2.1 via Harbor | Terminal tasks, re-audited items, and a harness that reruns cheaply at scale. |
| Resistance to overfitting over a contract term | SWE-bench Pro | A private held-out mirror is the only structural answer to a vendor tuning on your benchmark. |
| A number you will actually make a decision on | Your own 100-task set, plus a reference-solution run | Public benchmarks rank pairs of model and harness on someone else's repositories. |
The fifth row is the one that matters, and the mechanics of it — how to measure your own ceiling so harness rot stops reading as model regression — are in gold runs and the unsolvable task.
FAQ
Which of these has the highest scores right now?
That question has no portable answer, which is the finding rather than an evasion. Aggregator sites disagree about SWE-bench Verified by around twenty points because some report minimal bash-only agents and others report unconstrained commercial harnesses. Read the official board for the benchmark you care about, and only compare rows that name the same harness.
Is Terminal-Bench 2.1 less trustworthy because so many tasks needed fixing?
The opposite. Every benchmark built from real repositories accumulates the same defects; Terminal-Bench is the one that looked and published the count. A benchmark with no reported repairs has either not been audited or has not told you.
Does SWE-bench Pro's GPL licensing actually prevent contamination?
Unknown. It is Scale's stated reasoning for the choice, not a measured outcome, and no published study tests whether licence terms change corpus composition. The held-out private set is the part of Pro's design that works regardless.
Can I compare a Terminal-Bench 2.0 score with a 2.1 score?
No. Repairing broken and under-resourced tasks raises most agents, so the versions are different denominators. The run identifiers differ too, which is a useful accident.
How many repeats should I run?
Three to five, with a standard error printed next to the mean, is the prevailing practice among people who publish their methodology. Aggregate variance on these benchmarks is low but per-instance results move, so a single run is fine for a rough reading and useless for a decision between close systems.
Further reading
On this wiki:
- Gold Runs and the Unsolvable Task — how to measure your own eval set's ceiling.
- Benchmark Contamination — three different leaks that wear the same word.
- Benchmark Landscape (2026) — where these sit among the non-coding benchmarks.
- Eval Variance & Statistical Power — how many runs, and why pairing beats more runs.
- Reading Agent Benchmarks — harness sensitivity and why rank is not your task.
- The Cost of Evaluation — what repeated runs actually bill.
Project sources:
- SWE-bench and SWE-bench/experiments (leaderboard submissions)
- scaleapi/SWE-bench_Pro-os
- harbor-framework/harbor and terminal-bench-2-1
- SWE-rebench, An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (arXiv 2505.20411)