SWE-bench Verified saturated — five models within 0.7 points — and the field moved to SWE-bench Pro, HAL, Gaia2, and tau2-bench; Verified is now an audit signal, not a ranking.
SWE-bench Verified has five models within 0.7 points at the top and three Anthropic entries breaking away in the 93-95% band. The scoreboard stopped discriminating. The field moved to SWE-bench Pro (top ~59%), Gaia2 (best ~42% pass@1), tau2-bench with Telecom, and Princeton's HAL (cost-per-solve plus a 5-dim reliability dashboard). Reading agent benchmark headlines in 2026 requires knowing which of these each headline uses and what saturation looks like. This essay is that map.
SWE-bench Verified saturation.
SWE-bench Verified was the benchmark that broke agent-headline discipline in 2024-2025 — a curated subset of real GitHub issues where the "did the patch pass the tests" grader is unambiguous. It worked exactly long enough for the frontier labs to concentrate on it. By mid-2026 the top five entries land within 0.7 points of each other, with three Anthropic entries breaking away into the 93.9-95.5% band and the rest of the field clustered just below. A benchmark whose top five differ by less than the run-to-run noise of a single submission is not ranking anything; it is publishing a tie.
The correct read on Verified in 2026 is the one reading-benchmarks reaches for: treat it as an audit signal, not a leaderboard. A model landing at 45% on Verified is telling you something (it cannot do the basic form of the task); a model landing at 94% versus 93.5% is telling you nothing you can act on. The rank difference is smaller than the confidence interval on any single evaluation run. Attribution to specific model families in the top band is complicated by the "Anthropic sweep" pattern, which is directional evidence of benchmark familiarity as much as pure capability — a caveat that belongs on any 2026 slide that shows the Verified leaderboard.
SWE-bench Pro.
SWE-bench Pro is the successor Scale AI shipped in early 2026, and the reason it exists is exactly the saturation described above. Pro raises the difficulty ceiling by drawing from a harder distribution of real issues (longer patches, more cross-file coordination, more integration-test dependencies) and by tightening the grader (multi-test-file assertion, no partial credit for compilation-only fixes). The top of the Pro leaderboard sits around 59% for GPT-5.4-xHigh; the median frontier model is in the low 40s. That is a benchmark with room to move.
The other thing Pro did that matters is treat contamination as a legal deterrent rather than a hygiene problem. Test items are drawn from repositories under licenses that prohibit their use as training data without attribution; the benchmark's public position is that a model trained on Pro items is publishing a license violation. This is the mid-2026 pattern across new benchmarks: not "we hope you don't train on this" but "training on this creates a paper trail that survives your legal review." The effectiveness is unproven at scale but the direction is where the incentive gradient points.
Gaia2 and asynchronous agent eval.
Gaia2 measures something Verified and Pro cannot: what happens when the world changes between the agent's steps. Tasks are set in asynchronous environments — the underlying state (files, tickets, calendars, external services) mutates on its own schedule while the agent is planning — and success requires the agent to notice, adapt, and sometimes revert its own earlier actions. Best pass@1 is around 42% for GPT-5 high; most frontier models sit in the 25-35% band. The lower ceiling is honest: static benchmarks reward the shape of a solvable puzzle, and real deployment does not hand you a static puzzle.
The Gaia2 innovation worth naming is the write-action verifier. Every write action the agent takes (create a file, close a ticket, call an external API) is checked by a task-specific verifier that lives outside the agent's context and cannot be reasoned around. Verifiers know the temporal constraints of the environment ("this action was valid at t=0 but is invalid at t=30 because the underlying ticket was closed by another actor"), which is what makes the async setup gradable at all. The RAG eval essay makes a related point about assertion-side verification; Gaia2 pushes it into the action layer where most agent bugs actually live.
tau2-bench.
tau2-bench is Sierra's iteration on multi-turn customer-service benchmarking, and the piece worth reading is the Telecom subset. Tasks involve customer-support workflows with tool use, policy adherence, and voice-mode variants; grading is on outcome (was the ticket resolved), on policy compliance (did the agent follow the escalation rules), and on cost (how many turns and tool calls the resolution took). What tau2 catches that SWE-bench cannot is agents that resolve the surface task while violating a policy that only a policy-aware grader would notice — a class of failure that shows up in every production deployment and none of the code-benchmark leaderboards.
The 2026 version added Telecom and voice modes precisely because customer-support workloads are where enterprise agent adoption is landing hardest, and the earlier subsets skewed toward technical support flows that were easier for coding-optimized models. The Telecom subset is where the frontier-model ranking diverges most from SWE-bench Pro, because policy adherence and outcome-vs-compliance trade-offs do not correlate strongly with code capability.
HAL and cost-per-solve.
Princeton's Holistic Agent Leaderboard (HAL) is the piece the other four benchmarks have been missing: a single harness that reports cost-per-solve alongside pass rate, and a five-dimension reliability dashboard (consistency, predictability, robustness, safety, self-awareness) alongside the task score. HAL aggregates results across nine underlying benchmarks so a single model gets a portfolio number rather than a leaderboard rank on one axis, and the cost-per-solve figure ties everything back to a dollar number the eng team can actually plan against.
{
"model": "claude-opus-4.7",
"harness_version": "hal-2026.03",
"cost_per_solve_usd": 1.42,
"pass_at_1_mean": 0.61,
"reliability": {
"consistency": 0.88,
"predictability": 0.79,
"robustness": 0.71,
"safety": 0.94,
"self_awareness": 0.55
}
}
The report shape above is the one worth internalizing. A single pass-rate number for an agent is directional; a pair of "pass rate at cost $X" is planable; a reliability dashboard on top separates "same output every time" (high consistency, low self-awareness) from "notices when it is wrong" (moderate consistency, high self-awareness), which are utterly different production properties that a single number hides. The judge-calibration essay makes the parallel case for judge scores; HAL is the same discipline pushed into task-side benchmarks.
Contamination as legal deterrent.
Every modern benchmark has been quietly rebuilding its contamination story, and the 2026 pattern is licensing rather than obfuscation. The pre-2025 approach ("hide the test set, hope model trainers do not scrape it") failed at every scale it was tried; frontier model trainers have compute to crawl anything reachable and the incentive to. The 2026 approach uses non-permissive licensing on the underlying items so that training a model on them creates a documented license breach, which shows up in the trainer's own SBOM/data-lineage records. It is not a technical defense; it is a legal one, and legal defenses work on the margin the technical ones did not.
2026 agent benchmark headlines — snapshot
----------------------------------------------
SWE-bench Verified top-5 within 0.7 pts audit signal, not ranking
SWE-bench Pro top ~59% (GPT-5.4) harder distribution, tighter grader
Gaia2 best pass@1 ~42% async env, write-action verifiers
tau2-bench Telecom mixed by policy dim policy compliance + outcome + cost
HAL (Princeton) $/solve + 5-dim rel. portfolio across 9 benchmarks
----------------------------------------------
Contamination posture: licensing-based deterrent
Two takeaways for anyone reading benchmark headlines in 2026. First, name which benchmark the headline uses before believing the ranking, and if the answer is Verified, downgrade the ranking to an audit signal. Second, ask for the cost-per-solve number; if it is not published, assume the model that "leads" is priced two to four times its nearest competitor, because that is the modal pattern in the Anthropic-sweep entries at the top of Verified. A ranking without a cost axis is a ranking that a procurement team cannot use, and a benchmark culture that does not publish cost is a culture where the current leaderboard is worth less than it looks.