Eval Variance & Statistical Power

The number you shipped is a sample, not a measurement.

A deterministic test suite returns the same answer twice. An agent eval does not, and it fails to for at least four independent reasons, each of which contributes variance you will otherwise attribute to your change:

• Sampling. Any temperature above zero makes the trajectory a draw. And temperature zero does not save you — batched inference, mixture-of-experts routing and floating-point non-associativity all make identical inputs return different outputs, which is the subject of reproducibility and determinism.
• The environment. A tool times out, a page renders differently, a rate limit trips at step twelve. Long-horizon tasks accumulate these; the probability that at least one step hits a transient failure grows with the step count.
• The judge. If a model grades the output, the grade is itself a sample — and a drifting one. See judge calibration.
• Error compounding. This is the one specific to agents. A twenty-step task multiplies twenty step-level success probabilities, so small per-step variation produces large end-to-end variation. It is why agent evals are noisier than the single-turn evals people's intuitions were built on.

pass@k and pass^k answer opposite questions.

Two metrics, one character apart, pointing in opposite directions. pass@k is the probability that at least one of k attempts succeeds. It rises with k and it measures capability: can the model do this at all, given retries? pass^k is the probability that all k attempts succeed. It falls with k and it measures reliability: will this work every time a customer asks?

Which one you want follows directly from who consumes the output. If a human reviews every result and can retry, pass@k is the honest metric — you genuinely get k shots. If the agent acts unsupervised, pass@k is a fantasy, because your user gets one attempt and does not know they got the bad one.

The gap between them is not a rounding error. The τ-bench work that introduced pass^k for tool-agent tasks reported frontier models around 60% on a single attempt collapsing to roughly a quarter across eight — the same agent, the same tasks, two different questions. Drops of fifteen to twenty-five points between pass^1 and pass^4 are commonly reported across agent benchmarks. If your product promise is reliability and your dashboard shows pass@1, your dashboard is measuring something you do not sell.

Where the variance actually lives — and why more runs is the wrong lever.

Here is the part that reverses most teams' instincts. The variance in your headline score decomposes into two sources, and they do not respond to the same remedy. Write it out for a mean score over T tasks with R runs each:

Var(score)  ≈   σ²_between / T   +   σ²_within / (T · R)

σ²_between  variance across tasks    — some tasks are just harder
σ²_within   variance across runs     — the same task, different draws

Adding runs shrinks only the second term. The first term is divided by T alone and does not move no matter how many times you re-run. And on agent benchmarks the first term is usually the larger one, because task difficulty is enormously heterogeneous — a benchmark contains a handful of trivial instances, a long middle, and some that no current system solves. Sampling variation within a task is small next to the variation between a one-line config fix and a cross-module refactor.

The practical consequence: if you have budget for 500 more agent runs, spending it on 500 new tasks buys more precision than spending it on 5 extra runs across your existing 100. Teams reach for repeats because repeats feel like rigour, and then wonder why the interval barely moved. The interval barely moved because they attacked the term that was already small.

Two corollaries worth stating plainly. Resample tasks, not runs, when you bootstrap — the task is the unit of independent observation, and a bootstrap over runs will report an interval far too narrow. And when you cannot add tasks, at least stop pretending the ones you have are more informative than they are: a 50-task eval has a 95% interval of about ±14 points at mid-range scores, which is wide enough that it can only detect changes nobody needed statistics to see.

Pairing: the change that costs nothing and buys everything.

You almost never care about a score in the absolute. You care about a difference — new prompt versus old, model A versus model B, with and without the retriever. And a difference can be measured far more precisely than either of the two numbers going into it, provided you measure it correctly.

Run both systems on the same tasks, with the same seeds, and compare per task. Between-task variance — the dominant term from Step 3 — cancels out of the difference entirely, because a hard task is hard for both systems and its difficulty contributes equally to both scores. What remains is only the variance of the per-task differences, which is much smaller.

The arithmetic on a 500-task benchmark, the size of SWE-bench Verified:

• Unpaired. A score near 50% has a standard error of about 2.2 points, so a 95% interval of roughly ±4.4. Comparing two such scores independently, the difference needs to clear about 6 points before you can call it.
• Paired. Suppose the two systems agree on 460 of the 500 tasks and disagree on 40. Only the discordant pairs carry information. The standard error of the difference is about √40 / 500 = 1.3 points, giving a 95% interval near ±2.5.

Same data, same cost, and the detectable effect fell by more than half. This is McNemar's test, it is a hundred years old, and it is the correct instrument for exactly this shape of comparison. The wider practice of putting it in front of a merge is covered in eval-driven development and CI; this is the statistic that gate should be running.

How many runs, concretely.

The honest answer is that it depends on the effect you need to detect, but the defaults below are defensible and cheap enough to actually adopt:

• Five runs per task, minimum, at your production temperature. Below five you cannot estimate within-task variance at all, and you certainly cannot compute pass^k for any useful k. Five is not a lot of statistical power; it is the floor at which the numbers stop being uninterpretable.
• Report the median and the 5–95% range, not the best run. Reporting the maximum across runs is the most common form of accidental fraud in this field: it is a biased estimator that improves as you spend more, so it rewards budget rather than quality.
• Bootstrap over tasks with 10,000 resamples for the interval. It is a few seconds of CPU and it is not optional — a score without an interval is not a result.
• Prefer a paired design over any of the above. If forced to choose between five runs unpaired and one run paired with a fixed seed set, take the pairing.

Cost is the real constraint, and it is worth naming honestly: five runs across 500 tasks is 2,500 agent trajectories, which at agent prices is a real budget line. Two mitigations that work. First, tier your suite — a small fast subset gates every pull request, the full paired comparison runs nightly or per release. Second, remember that cost per completed task is itself a number you should be tracking, and a cheaper configuration that needs three attempts is not cheaper. Princeton's HAL harness makes cost-per-solve a first-class axis for exactly this reason; see HAL and asynchronous agent eval.

The decision rule, and the failure modes it prevents.

Turn all of the above into one rule your team can apply without re-deriving it: do not act on a delta smaller than your interval's half-width. Not "note it as a promising trend", not "ship it since it's directionally positive" — do not act. If the difference matters enough to act on, it matters enough to measure until it clears the bar, and if it never clears the bar with a paired design on your full suite, it was not there.

Three failure modes this rule kills outright:

• Ratcheting on noise. Accept every change with a positive delta and you accept roughly half of all no-op changes, then bake them in permanently. A codebase full of prompt tweaks that each "improved things by two points" is a codebase full of accumulated coin flips.
• The re-run until green. If a gate fails and someone re-runs it until it passes, the gate now measures persistence. Fix the seed set, run once, and treat the result as final — the re-run must be a deliberate, logged decision, not a button.
• Comparing across incomparable runs. Last month's number was produced against a model snapshot that no longer exists. It is not a baseline; it is a memory. Re-run the baseline in the same batch as the candidate, always.

Related: trajectory and process evaluation for what to score beyond the final answer, reading agent benchmarks for applying this scepticism to someone else's numbers, and evals 101 for the foundations.