When the Check Has to Act

Step 1Two kinds of verifier, and only one of them is passive.

A passive verifier reads state that already exists. It opens a database, hashes a file, parses the response body — operations with no side effects and essentially no failure mode of their own beyond "the artifact is missing," which is itself a legitimate fail. Unit tests are passive. So is scoring a model's final text against a gold answer.

An active verifier has to change the system before it can observe it. It must focus a window to read the title bar, flush an application's in-memory buffer to disk before the bytes exist, re-run a query because the result was never persisted, or navigate to a settings page to see whether a toggle moved. The moment your environment is a real desktop, browser, or phone rather than a function call, most interesting checks are active ones, because applications hold state in memory that the filesystem does not have yet.

Here is a real one, from OSWorld's LibreOffice Calc suite — a task that asks the agent to add a Maturity Date column. The evaluator block runs before any comparison happens:

# OSWorld task 4172ea6e — the evaluator's postconfig, in order
activate_window  window_name: "MaturityDate.xlsx - LibreOffice Calc", strict: true
sleep            0.5
execute          python -c 'import pyautogui; pyautogui.hotkey("ctrl","s")'
sleep            0.5
# only now:
func             compare_table
expected         cloud_file  # fetched from a HuggingFace dataset repo
result           vm_file     /home/user/MaturityDate.xlsx

Four things must go right before the agent's work is even looked at. The window title must match exactly — and the agent may have legitimately renamed or re-saved the file. The save dialog must not appear, or the keystroke lands in it. Half a second must be enough for LibreOffice to finish writing. And the gold file must download. A correct agent fails this task on any one of them.

Step 2The failure rate is invisible because it is reported in the agent's column.

This is the part that costs teams months. An active verifier's failures are not logged as infrastructure errors; they are logged as task failures, which is the same thing a genuine agent mistake produces. A score of 38% contains three populations — tasks the agent got right, tasks the agent got wrong, and tasks nobody can speak to — and the third is silently folded into the second.

The direction of the bias is worth being precise about, because it is not simply "scores are too low." Active-verifier noise is anti-correlated with agent capability in a way that flatters bad agents and punishes good ones. A verifier step only runs if the harness reaches it, and an agent that crashes out at step three never gets to the postconfig at all — it fails cleanly, for a real reason. The agent that actually completes the spreadsheet is the one exposed to the Ctrl+S race, the focus check, and the download. The closer a system gets to finishing tasks, the larger the share of its remaining failures belong to the grader.

Scale matters here too, because the per-step risks compound. Across OSWorld's task files the setup and evaluator stages use execute 358 times, sleep 307 times, launch 303 times, download 293 times and activate_window 200 times, against 368 tasks in test_all.json. That is not an exotic edge of the suite; it is the suite. A benchmark whose grading path issues several hundred timed GUI operations and several hundred network fetches has a floor on its own reliability, and no published score tells you where that floor is.

Step 3The gold-state self-test: run the check against a known-good machine.

The fix is cheap and almost nobody ships it. For every task, you need a recorded end state that is definitionally correct — the saved .xlsx, the configured app, the submitted form — and a mode that restores that state and then runs the verifier against it, with no agent in the loop at all.

The verifier must pass. If it does not, the task is broken, and you have learned that before attributing anything to a model.

# one task, two runs, three outcomes

run_agent(task)    -> artifact_A
restore(task.gold) -> artifact_G   # the known-good end state

verify(artifact_G) == FAIL  -> INDETERMINATE   # the check is broken; drop the task
verify(artifact_G) == PASS  -> trust the check, then:
    verify(artifact_A) == PASS -> PASS
    verify(artifact_A) == FAIL -> FAIL              # now attributable to the agent

# report all three. denominator = PASS + FAIL, never PASS + FAIL + INDETERMINATE.

Two properties make this worth the engineering. First, it is a per-run check, not a one-off audit: the gold replay happens on the same VM image, the same day, behind the same network as the scored run, so it catches the CDN that went down this morning and the image update that changed a window title last week. Second, it localises blame without needing a human. A task whose gold replay fails is removed from the denominator and filed as a bug against the suite — which is how an eval set stays a measurement instead of slowly becoming a pessimistic constant.

The cost is one extra environment-run per task per sweep, which roughly doubles the infrastructure bill of the cheapest sweep you run and is a rounding error on an expensive one. Amortise it: the gold replay does not need to run on every model you compare, only once per environment image per day, because it is testing the environment and not the model. Account for it the way you account for sampling in budget-matched evaluation — as a fixed cost of getting a number you can defend.

Step 4Three outcomes, not two — and the third one goes in the report.

The reason most harnesses have two outcomes is that reward is a float and there is nowhere to put "unknown." BrowserGym's task interface is explicit about this: validate(page, chat_messages) returns (reward, done, message, info), and a verifier that cannot decide has exactly one way to say so, which is to return 0.0 and look identical to a failure. AndroidWorld has the same shape — is_successful(env) -> float. Neither is badly designed; a scalar reward is the right interface for training. It is the wrong interface for reporting, and the two get conflated because the same function serves both.

So carry the third state out of band, in the info dictionary that is already there, and surface it in the summary. A defensible agent result looks like this:

Suite: osworld-verified      Image: ubuntu-2026-10-14      Sweep: 3 seeds

  attempted            368
  indeterminate         17   # gold replay failed; excluded
  scored               351
    pass               134
    fail               217

  success rate       38.2%   (134/351, 95% CI 33.1-43.5)
  NOT               36.4%   (134/368) — the number a two-outcome harness prints

Those two percentages differ by 1.8 points on a 4.6% indeterminate rate, which sounds survivable until you compare two systems measured a week apart on different image builds. The reported gap between them then includes the difference in their graders' health, and that difference has no reason to be small or stable. This is the same identification problem as an unreported user simulator in multi-turn evaluation: the measurement contains a second system you did not intend to evaluate, and omitting it does not make it absent, only unnamed.

Step 5What transfers to your own product, where there is no benchmark.

Nothing above is specific to desktop benchmarks; they are just where the problem is legible because the code is public. The same structure appears the moment your own definition of done requires an action to observe. "Did the invoice get posted?" is answered by a query against a ledger whose read replica lags. "Did the ticket get escalated?" is answered by an API that rate-limits your checker before it rate-limits your agent. "Did the file get written?" is answered by a storage layer that is eventually consistent, so a check that runs 200 ms after the agent finishes is measuring propagation delay and calling it agent failure.

Three rules carry over cleanly:

• Write the checker before the agent, and test it against a hand-made success. If you cannot produce the correct end state by hand and have your checker confirm it, you do not yet have a definition of done — you have a hope. This is the single highest-return half-day in setting up an agent evaluation, and it is almost always skipped because the checker feels like test infrastructure rather than product.
• Prefer the passive read, even when it is weaker. A check that reads an append-only audit log is less expressive than one that re-runs the agent's query, and it is worth the loss: it has no side effects, no focus, no timing, and no failure mode that isn't also a real failure. Where you can arrange for the agent's own action to leave a durable artifact, you convert an active verifier into a passive one, which is the cheapest reliability win available in agent evaluation.
• Never let an indeterminate run silently become a failure. Emit it as its own class and alert on its rate. A rising indeterminate rate is the earliest signal that your environment drifted, and it arrives weeks before anyone notices the quality regression it will be mistaken for — which is exactly the misattribution a failure taxonomy exists to prevent.

Step 6The honest trade-off.

Active verification is not a mistake to be designed away. Some tasks genuinely cannot be scored without touching the machine, and refusing to write those checks means refusing to measure the work agents are actually sent to do — which is worse than measuring it noisily. The error is not the active check; it is the two-outcome report that hides its failures inside the agent's.

The open question nobody has answered well is how to price an indeterminate run against a wrong one. A suite that drops 5% of tasks for grader failure and a suite that scores them all with 5% grader noise produce different numbers, and the first is more honest while the second is better powered. Published agent benchmarks mostly do neither; they report one rate over all attempted tasks and leave you to assume the grader was perfect. Until that changes, assume a floor of a few points of grader noise under any desktop or browser agent figure you read, and treat differences smaller than that as unresolved — the same discipline reading benchmarks asks for, applied one layer lower than usual.