Gold Runs & the Unsolvable Task

Step 1Three ways a task dies while you are not looking.

These are not exotic failures. They are the ordinary consequence of building an eval set out of real software, and they have different detection signatures.

• The environment rots. A dependency resolver picks a newer transitive version that breaks an import. A registry mirror goes slow. A base image tag you pinned moves underneath you. An external service the task touches changes a response shape. None of this produces an error that says "this task is now invalid" — it produces a failing test, which is indistinguishable from an agent that got the answer wrong.
• The budget stops fitting. The task is still solvable, but not inside the wall-clock limit, token ceiling, memory cap or step count the harness grants. This is the most insidious of the three because the task passes for a sufficiently fast agent and fails for a slower one, so it reads as a capability difference. It is the category Terminal-Bench's maintainers counted separately, and they found eight such tasks in eighty-nine.
• The specification drifts out of alignment with the check. The instructions and the tests agreed when the task was written. Then someone tightened a test, or the upstream project renamed an interface the instructions still describe, and now the only way to pass is to do something the prompt does not ask for. Nothing in the repository records this divergence.

Step 2The reference run is the only measurement that finds them — and it costs compute, not tokens.

A gold run (or reference run) is a pass over the whole suite in which the known-correct answer is applied in place of the agent: the reference patch for a code task, the recorded correct tool sequence for an agentic one, the labelled answer for a classification task. Everything else is identical — same container, same limits, same scorer, same day. The gold run answers one question: of my N tasks, how many are currently passable at all?

Three properties make this affordable, and they are why the practice is underused rather than expensive.

• No model is in the loop. The reference solution is a fixed artefact, so a gold run bills container seconds and zero inference. On a suite where an agent run costs tens of dollars in tokens, the gold run is a rounding error — see the cost of evaluation for where the money actually goes.
• It is deterministic, which means a gold failure is a real signal rather than a sample. If the reference patch failed, the task is broken; there is no "try five times" question to argue about, unlike the agent-side variance discussed in eval variance and statistical power.
• It parallelises perfectly and can be scoped. A full gold run on a schedule, plus a targeted gold run on any task whose image digest, lockfile or scorer changed since the last green pass, gets you most of the coverage for a fraction of the time.

# one cycle, two sentinels, one measurement
gold   = run(suite, solver=reference_patch)   # upper bound: what is passable
null   = run(suite, solver=no_op)             # lower bound: what passes for free
agent  = run(suite, solver=system_under_test)

achievable = gold.passed - null.passed      # tasks that both work AND discriminate
score      = agent.passed / achievable        # the only fraction worth reporting

# gold costs container time and no model tokens — it is the cheapest run you make

Step 3What the denominator does to your numbers.

Work it through once with concrete figures, because the size of the effect is the argument.

A suite of 180 tasks. On the day you build it, the gold run passes 180 of 180, and your candidate agent passes 112 — a score of 62.2%, which goes on the dashboard.

Six weeks later you re-run the same agent, unchanged, on the same suite. It passes 104, which is 57.8%. That is a 4.4-point drop, and in most organisations it triggers a model-regression investigation, a rollback conversation, or a vendor email.

Now run gold. It passes 168 of 180. Twelve tasks have died: nine where a transitive dependency stopped resolving, two where a base image tag moved, one where a 600-second cap no longer fits a slower registry mirror. Your agent had been passing eight of those twelve, which is exactly the eight passes it lost.

Against the achievable denominator, the agent scored 104 / 168 = 61.9% — three tenths of a point from where it started. Nothing regressed. You spent a week investigating a measuring instrument.

Step 4The bias is one-way, so one sentinel is not enough.

This is the part that is genuinely non-obvious. Environment rot can only remove passes, never add them, so it is not noise — it is a drift with a sign. Over a year of unmeasured maintenance, an eval set reports a slow decline that no model caused, and every real improvement is netted against it. Teams that live with this long enough learn to distrust their own evals, which is a worse outcome than having none.

A gold run catches that direction and is blind to the other one. The opposite failure — a task that becomes passable without doing the work — inflates your score and leaves the gold run perfectly green, because the reference patch still passes too. It happens when a test is weakened to unblock a flaky build, when the fix lands in the base image so the repository already contains it, or when a scorer change makes a substring match sufficient. The adversarial version of this is the subject of eval integrity and scorer gaming; the accidental version is more common and nobody is looking for it.

So run a null run as well: the same suite with a solver that does nothing — an empty patch, a no-op tool sequence, a refusal. The invariant is that a null run passes zero tasks. Any task it passes is a task that has stopped discriminating, and it should be quarantined rather than counted. Two sentinels bound your suite from both sides, and the arithmetic in STEP 2 subtracts the null passes out of the denominator so a broken-open task cannot flatter you.

• A cheap third sentinel, if you can afford it: a deliberately wrong solver — the reference patch with one line inverted, or the correct tool call with a wrong argument. It should fail everywhere. Where it passes, your scorer is too loose for that task specifically.
• Pin by digest, not by tag. Most environment rot enters through a tag that resolved to something new; this is the same hazard unpinned vendor defaults describes on the production side, and the remedy is identical.
• Treat the resource budget as part of the task specification and version it with the task. A timeout you tuned on fast hardware is a hidden parameter of your benchmark; changing the runner changes the score. See budget-matched evaluation for the comparison side of the same point.

Step 5Roughly three in ten is the published rate, not a worst case.

The only maintainer team that has published a measurement of its own task validity is Terminal-Bench's, and the number is the reason to take this seriously rather than file it as hygiene.

Terminal-Bench 2.0 shipped in November 2025 with 89 tasks, selected from 229 candidates contributed by 93 people and reviewed by three experienced humans — a 39% acceptance rate, which is about as careful as hand-built eval sets get. Six months later the maintainers released 2.1 to fix issues in 28 of those 89 tasks (the 2.1 repository's own README says 26 were modified, so the honest figure is a 26–28 range from two maintainer sources). Nine had external dependencies that changed after the benchmark was built. Eight had resource budgets too small for at least one valid approach. A third group, uncounted in the announcement, had instructions that were no longer aligned to their tests.

Three details are worth extracting:

• The categories match STEP 1 exactly, which is evidence that the taxonomy is the real one rather than a tidy invention. Dependency drift, budget drift, specification drift.
• Expert curation did not prevent it. A 39% acceptance rate and three human reviewers per task bought a better set, not a durable one. Decay is a function of elapsed time and external change, not of how carefully you wrote the task.
• The claim they could then make is the valuable artefact. After the repairs, the maintainers stated that no task in 2.1 is unsolved. That is a sentence about the ceiling, and it is the sentence your own dashboard should be able to produce. Note also that it makes 2.0 and 2.1 scores non-comparable — repairing broken tasks raises most agents — which is the price of honesty about a denominator and worth paying.

The infrastructure the field has built around this points the same way. SWE-bench moved to a fully containerised Docker harness in mid-2024 and now recommends roughly 120 GB of free storage to run locally; SWE-rebench ships a pre-built image per instance. Both are expensive answers to the question "why did this task stop working", and both are cheaper than the alternative. The broader lifecycle — replacement rates, holdouts, re-baselining — is eval-set maintenance; this page is about the one measurement that tells you when the lifecycle has fallen behind.

Step 6Report three states, and alarm on the third.

The reporting change is small and it is where the value lands. A task has three outcomes, not two.

• Pass — the agent solved it and the task was passable.
• Fail — the agent did not solve it and the task was passable.
• Unavailable — the reference solution did not pass, or the null solver did. The task is excluded from both numerator and denominator for this cycle.

The third state is not a new idea — when the check has to act argues for exactly the same escape hatch when the verifier is the thing that failed. The point here is that it needs a second trigger: a task can be unavailable because the grader broke, and it can be unavailable because the task itself is dead, and only a reference run distinguishes them.

Then publish three numbers together: the score against the achievable denominator, the achievable count itself, and the unavailable rate. A score of 61.9% over 168 of 180 tasks with 12 unavailable is a legible result. "57.8%" is a number that has silently merged a model measurement with an infrastructure fault.

• Alarm on the unavailable rate crossing a threshold, not on the score. A suite that goes from 2% to 9% unavailable in a week has had an infrastructure event, and it will show up as a quality regression if you let it.
• Alarm on the absence of a gold run. A missing or stale gold result must not default to "assume 100% achievable" — that silently restores the original bug. A frozen ceiling looks exactly like a healthy one.
• Never silently drop an unavailable task. Dropping it from the denominator without recording it is how a suite shrinks from 180 to 120 over two years while the headline score drifts upward, which is a real and common failure.
• Re-baseline after a repair. Fixing twelve tasks changes the denominator, so the series before and after is not continuous. Record the version, as reproducibility and determinism would for any other measurement, and compare within a version.

Related: benchmark contamination for the leak that inflates the numerator rather than shrinking the denominator, evaluating against live systems for the case where the environment is not yours to pin, and unsatisfiable tasks for the agent-facing version of the same problem.