65% Once, 25% Twenty Times: Your Headline Score Is Mostly Flake

What Thinkingbox measured

Thinkingbox is a sandbox and benchmark for agents working in stateful business workflows, released on 19 August 2026 by researchers at Microsoft with collaborators at the University of Pittsburgh, Northwestern University and UC Irvine, under the paper title One Success Isn't Reliability (arXiv:2608.19741). The framework is on GitHub under an MIT licence; tool mocks are defined as MCP servers, so the agent under test talks to the sandbox the same way it talks to production.

Thinkingbox-bench, the executable suite that ships with it, contains 507 policy-conditioned workflows across five domains — retail, hospitality, auto insurance, neobank internal IT, and consulting IT and HR support. "Policy-conditioned" is doing real work in that sentence: each task hands the agent a written business policy and the correct answer depends on applying it, so a task can be failed by doing something competent that the policy forbids.

The grading is the second thing worth noting. Outcomes are checked against the terminal state of the backend the agent was manipulating, plus collateral effects — what else changed that should not have. The agent's own account of what it did is not the artefact under test. That distinction turns out to explain a large share of the failures: the paper's recurring pattern is a plausible, well-formed response sitting on top of an incorrect state change, or a failure the agent recovered from badly and then reported as handled.

The metric itself is not new. pass^k — succeed on all k trials, as opposed to pass@k's at least one — was introduced with τ-bench in 2024, which reported the same collapse: an agent scoring around 60% on a single attempt fell to roughly 25% at pass^8 on the retail split. Two years later the headline number in nearly every model announcement is still the single attempt.

Run the arithmetic and the score falls apart

Take the two published numbers seriously for a moment. If the agent had a flat 65.36% chance on every attempt at every task, drawn fresh each time, then twenty successes in a row would occur with probability 0.6536²⁰ ≈ 0.0002. A benchmark of 507 tasks would show roughly zero of them passing twenty out of twenty. The measured figure is 25.25%.

Three orders of magnitude of disagreement is not noise, and it has exactly one explanation: outcomes are strongly correlated within a task. Whatever varies between runs is not sprinkled evenly over the suite. Some tasks the agent simply gets right, run after run. Others it gets right sometimes.

You can put a rough shape on that split with one assumption — that a task the agent is genuinely flaky on essentially never survives twenty consecutive attempts, which holds for any per-attempt rate below about 0.8. Then the deterministic band is the pass^20 figure, 25.25% of tasks, and the per-attempt success rate on everything else falls out of the pass@1 total:

deterministic band  d = pass^20              = 0.2525
remainder           r = 1 - d                 = 0.7475
per-attempt rate    q = (pass@1 - d) / r
                      = (0.6536 - 0.2525) / 0.7475
                      = 0.537

So: on a quarter of the suite the agent is reliable, and on the other three-quarters it succeeds about 54% of the time. That is a two-population caricature of what is really a continuum, and the real distribution certainly includes tasks the agent never solves at all, which this fit folds into the same bucket. But the direction is not in doubt, and it is the direction that matters. Forty of the sixty-five points in the headline score are contributed by tasks the agent solves sometimes.

"Sometimes" is operationally worse than "never"

The intuition to overturn is that a 54% task is halfway to a good task. It is not. In deployment terms it is worse than a task the agent fails every time, and the reason is that failure modes get handled in proportion to how visible they are.

A task the agent never completes announces itself on day one. Someone tries it in staging, watches it fail, and the system grows the thing it needed anyway: a fallback, a human queue, a refusal, a scope boundary in the product copy. The capability gap becomes a design decision, which is the correct outcome.

A task the agent completes half the time announces nothing. It passes the demo. It passes the eval — because the eval ran it once, and one sample from a 54% coin comes up heads more often than not. It ships. Then it fails for roughly half the users who hit that path, in a way that is irreproducible for whoever is on call, and it is triaged as a flake rather than as a missing capability. Meanwhile every dashboard your team built is a mean over runs, and a mean cannot distinguish "works for everyone, degraded" from "works perfectly for half of them".

The compounding is the part that turns this from annoying into disqualifying. Business workflows are chains. A five-step process where each step is drawn from that 54% band completes end to end 4.5% of the time — and a workflow assembled from steps that each demoed fine is exactly how teams arrive at a pilot that works in the room and dies in the quarter. This is the mechanism underneath the pattern our earlier post on the agent pullback described from the outside: the pilots did not fail because the models were incapable. They failed because capability was measured once.

There is also a procurement consequence. Two models can report the same pass@1 with completely different deployment value — one deterministic on 60% of tasks and hopeless on the rest, the other a 65% coin on everything. The first is a product. The second is a demo. No leaderboard column you are looking at separates them.

Grade the state, not the sentence

Thinkingbox's second contribution is quieter than the reliability gap and easier to act on this week. The suite grades the backend state the agent left behind, and it grades collateral effects — the records it touched that no correct execution would have touched.

Compare that with the default. Most agent evals grade a transcript, usually with a judge model reading the final message. The transcript is the one artefact the agent under test fully authors. When a model has been trained to produce well-formed, confident summaries of its own work, grading the summary measures the thing the model is best at, and the failure the paper keeps surfacing — a plausible response over an incorrect state change — is precisely the failure that survives that grader intact.

The database is not persuadable. Neither is the ledger, the ticket queue, the calendar, or the row count. If your agent writes anywhere, you can assert on what it wrote, and those assertions do not drift when the model changes its prose style. The general argument for this sits in outcome vs trajectory evaluation; Thinkingbox is the strongest empirical case for it we have seen, because it isolates a failure class that only a state check catches.

Collateral effects deserve their own line. An agent that resolves the ticket correctly and also cancels an unrelated subscription has passed every outcome assertion you thought to write. Snapshot the world before and after, diff it, and treat any change outside the expected set as a failure — the eval equivalent of the side-effect ledger that idempotency and retries argues for at runtime.

What to change in your own suite

Run every eval task k times and report pass^k next to pass@1. You do not need twenty. At k=5 a deterministic task and a 54% task separate cleanly — the coin survives five in a row 4.5% of the time — and five runs of a forty-task suite is an affordable nightly job. Report both numbers; a single number that hides a bimodal distribution is not a summary, it is a loss of information.

Bucket the suite into always / sometimes / never, and make the middle bucket the roadmap. This is the artefact the exercise exists to produce. The never bucket is a scope conversation. The always bucket is what you are allowed to promise. The sometimes bucket is engineering work with a known address — and it is usually addressable, because per-task flakiness generally traces to something concrete: an ambiguous tool description, a policy the model has to hold across too many turns, a retrieval step that returns different context on different runs, a schema the model fills in two different ways.

Do not let a fix be judged by the same single-sample method that missed the bug. Re-run the fixed task k times. A change that moves a task from 54% to 71% looks like a fix under one sample and is still a coin. This is where eval variance and statistical power stops being a statistics lecture and starts being the difference between shipping a fix and shipping a rumour.

Assert on state, and snapshot for collateral damage. If your agent writes to anything, write assertions against the write. Keep a judge for the parts that are genuinely about language — tone, refusal quality, explanation — and stop using one where a query would do.

Use pass^k as the model-swap gate. When a new model arrives, the interesting question is not whether the headline moved. It is whether tasks migrated between buckets — and models with near-identical pass@1 routinely differ on which tasks they are deterministic about. Freeze the suite, run both at the same k, diff the buckets. Our comparison of the four open-source eval frameworks covers which harness makes repeated-trial runs cheap enough to do on every model change.

None of this requires adopting Thinkingbox. The suite is worth reading for its task design, and the MIT licence makes the scenario format a reasonable thing to borrow. But the finding transfers without the tool: you already have a benchmark, you are already reporting one number from it, and the second number costs you k times the compute and tells you which half of the first one was real.

Further reading

On this wiki:

• Outcome vs trajectory evaluation — what to grade when the transcript is authored by the thing under test.
• Eval variance & statistical power — how many runs a claimed improvement actually needs.
• Quality regression detection — catching the drop before your users report it.
• Reading benchmarks — the general skill of asking what a score was measured over.

Sources:

• One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
• microsoft/thinkingbox on GitHub
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains