The tasks were public, so were the defects

At a glance

Epoch AI's Benchmark Reviews programme applies a published rubric to benchmarks the industry actually quotes, and issues one of three verdicts. The registry keeps growing, so read it rather than quoting this table — these are the counts most widely reported in mid-September 2026.

What "Flawed" actually asserts

The verdict is narrower and more useful than the word suggests, and the threshold is published: a benchmark is rated Flawed when scoring defects could affect at least 20% of the full score. For a suite of equally weighted, all-or-nothing tasks, that reduces to a count — at least a fifth of the tasks carry a defect. Failing any item in the rubric produces the verdict.

Notice what that does and does not say. It does not say the models are worse than reported, or that the ranking is inverted, or that the benchmark is worthless. It says the measurement has an error term from the scoring apparatus itself, and that the error term is large enough to be comparable to the differences people read off the leaderboard. That is a claim about resolution, not about direction. A benchmark can be Flawed and still be the best instrument available for its domain; what it cannot do is support a two-point distinction between two model names.

Terminal-Bench 4.0.0: 30 of 66

The clearest case is the coding-agent benchmark the field moved to during 2026. Terminal-Bench 4.0.0 was a maintenance release: it removed 8 tasks, updated 20, and recalibrated per-task resource limits relative to 3.0, leaving 66 tasks. The removals were the saturated ones — tasks frontier agents had mostly solved, which therefore no longer separated anybody. On its face that is exactly the right maintenance to do, and it is why the version exists.

Epoch's review, dated 4 September 2026, reports publicly noted scoring defects in 30 of those 66 tasks — about 45.5%, more than twice the Flawed threshold. The review also records that the tasks, verifiers, reference solutions and resource settings are all disclosed. Those two facts belong in the same sentence: the benchmark is Flawed and fully transparent, and it is Flawed because it is transparent enough for anyone to check.

SWE-bench Verified received the same verdict in a review dated 3 September 2026, which is worth sitting with for a moment. "Verified" in that benchmark's name refers to human screening of whether each issue was solvable and well specified — a real and valuable curation step. It was never a claim that the tests accompanying each instance were strong enough to distinguish a correct patch from a plausible wrong one, and the published research on that question says they frequently are not. We went through those numbers in patch generation and test-driven loops: on SWE-bench Verified, 7.8% of patches counted as correct while failing the developers' own test suite, and differential testing put 29.6% of plausible patches at behaviour differing from the ground-truth fix.

The disclosure paradox

Here is the part that should be uncomfortable. Benchmarks that disclose everything are the ones that can be rated Flawed. Benchmarks that disclose little enough get "Not enough information" — a verdict that reads softer and is strictly worse, because it means the instrument cannot be checked at all. The incentive this creates is obvious and bad, and the only defence against it is that readers understand which of the two is the actual warning.

It also reframes who failed. Epoch did not run a costly investigation and surface something hidden; it read tasks and verifiers that had been on the internet the entire time, and found defects that other people had already noted publicly. The information needed to put an error bar on the numbers in every model announcement of the past year was free and available. It was not used, because reading a benchmark is slow, unrewarded work and quoting its number takes one sentence. The same asymmetry runs through the eval literature — Epoch reissued its own FrontierMath in June 2026 after addressing errors in 42% of the problems, and its analysis of GPQA Diamond argues that at least 90% of that benchmark is valid, which is a positive finding precisely because someone bothered to establish it.

The useful generalisation is not "benchmarks are broken". It is that a benchmark is a piece of software with no test suite of its own, maintained by people with less funding than the labs quoting it, and that the field had been granting it an authority nobody had earned on its behalf. This is the structural point that eval integrity and scorer gaming and benchmark contamination approach from two other directions.

Two independent ways your quoted number is wrong

Stack this finding against the variance work from the same season and the picture is worse than either on its own. A Stanford and UIUC decomposition across 22 agent benchmarks found that ranking a fixed model-and-scaffold pair is reliable while ranking the underlying model from the same rollouts often is not — the argument in the leaderboard ranks the pair, not the model. That is a statement about the third box in the diagram: the run.

Epoch's verdict is a statement about the second box: the task set. The two error sources are independent, and the usual remedies do not touch each other. Running more samples narrows variance and does nothing about a defective verifier; auditing the tasks fixes the verifier and leaves the scaffold confound exactly where it was. A number quoted without both qualifications is carrying two unstated error terms, and the leads being reported between frontier systems are routinely smaller than either one.

What to do with a score from a Flawed benchmark

Not discard it. The practical adjustment is three habits, none of which require you to run anything.

Quote the verdict with the score. "68% on Terminal-Bench 4.0.0 (Epoch: Flawed, defects in 30 of 66 tasks)" is the honest form, and it is barely longer. Internally, make the benchmark registry a field in whatever document records a model decision, next to the number — the same discipline as recording a library version.

Treat the defect share as a floor on the resolution. If scoring defects could move 20% of the score, differences smaller than that are not measurements. In practice this retires most published model-versus-model comparisons on these suites and leaves the large gaps, which were the only ones worth acting on anyway.

Pin the version, in the citation and in your own pipeline. Terminal-Bench moved from 2.1 to 3.0 to 4.0.0 with tasks removed and resource limits recalibrated at each step; a score is meaningless without the version, and a time series across versions is not a time series. This is the same failure mode as an unpinned dependency, and it has the same fix.

And the durable move, which has not changed: a small eval set built from your own traffic beats every public number for your decision, because it is the only one measuring your distribution. The public suites are for orienting yourself in the field; reading benchmarks critically and eval set maintenance are where the work that actually informs a deployment happens.

Further reading

On this wiki:

• Patch generation and test-driven loops — the same weakness inside your own repository, and how to measure it.
• Eval integrity and scorer gaming — when the scorer is the attack surface.
• Benchmark contamination — the other reason a public score drifts away from capability.
• The leaderboard ranks the pair, not the model — the variance half of the picture.
• Reading benchmarks critically — the short version for anyone about to quote a number.

Sources:

• Epoch AI — Benchmark Reviews methodology
• Epoch AI — Terminal-Bench 4.0.0 review
• Epoch AI — SWE-bench Verified review
• Terminal-Bench 4.0 release notes