The sentence everybody reads off an agent leaderboard — “this model is better than that one” — is the one claim the leaderboard supports worst, and a September 2026 variance decomposition of 22 agent benchmarks now puts a number on how much worse. Ranking the system scores 0.935 to 0.994 on reliability; ranking the model behind it scores 0.148 to 0.841. The fix is not more tasks. More tasks is capped.
At a glance
The paper is Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard (arXiv:2610.00651), by Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer and Sanmi Koyejo, submitted 30 September 2026. It borrows Generalizability theory from psychometrics and builds a Bayesian variance decomposition that tolerates the sparse, imbalanced designs real agent leaderboards actually have.
| Claim being made | Reliability (Eρ²) | What that means in practice |
|---|---|---|
| Ranking a fixed model + scaffold system | 0.935 – 0.994 | Dependable: re-run the leaderboard and the order holds |
| Ranking the underlying model | 0.148 – 0.841 | At the low end, the ordering is close to arbitrary |
| Adding unlimited similar tasks | at most +0.097 | Where scaffold coverage dominates, the usual remedy is capped |
The data it decomposes is the public record: 22 benchmarks drawn from the Holistic Agent Leaderboard and the Harbor Index. HAL is the standardised harness from Kapoor et al. at ICLR 2026, validated over 21,730 agent rollouts across nine models and nine benchmarks. The Harbor Index is a meta-dataset of task-level scores from 29 agent benchmarks across nine LLMs and four scaffolds — which is exactly the shape of data you need to ask whether the scaffold is doing the ranking.
Two claims, one table, wildly different reliability
A generalizability coefficient answers a blunt question: if you ran this whole measurement again under equivalent conditions, how much of the ordering would survive? At 0.99 essentially all of it. At 0.15 you are reading noise with a ranking printed on it.
The top bar is the good news, and it is genuinely good news. Agent leaderboards are not broken. If the thing you are choosing between is “this agent, as built, running this harness” and “that agent, as built, running that harness,” the ranking is about as dependable as a measurement of this kind gets. Teams picking a product get a usable answer.
The bottom bar is what happens when you strip the scaffold off and talk about the model. That is the move everyone makes: a leaderboard row says a model name, a procurement deck quotes the model name, and the scaffold — the prompt, the tool definitions, the retry policy, the step limit, the harness — disappears into a methodology footnote. The paper's contribution is to show that the discarded part was carrying a large and variable share of the signal, and to name the measurement it implies: inter-scaffold reliability, which asks directly whether swapping scaffolds preserves the ranking of models. The answer across these evaluations is that scaffold effects vary considerably.
Why “add more tasks” hits a wall
The conventional response to a noisy benchmark is to make it bigger, and for one species of noise that is exactly right: sampling error over tasks shrinks as you add tasks. That advice has been the standing recommendation here too, on eval variance and statistical power, where the argument is that 500 new tasks buy more precision than five extra runs over the tasks you already have. That remains true for the variance it addresses. This paper identifies the term it cannot touch.
The headline quantity is the cap: where scaffold coverage is the dominant source of uncertainty, even an infinite number of similarly built tasks raises model-ranking reliability by at most 0.097. An unbounded budget, spent the usual way, moves the number by less than a tenth. That is the whole argument for re-reading the word “similarly” as a load-bearing one — tasks drawn from the same generator vary in the dimension you are already sampling well and are identical in the dimension you are not sampling at all.
The constructive half follows: broaden the measurement conditions rather than deepen one of them. A secondary summary of the work reports that pooling diverse benchmarks lifts projected cross-task reliability from about 0.44 to 0.75 at the same task budget — same spend, different allocation. Treat that specific pair of numbers as a reported figure rather than one we have traced into the paper's tables; the direction is the part the abstract supports on its own.
What this does to the sentences people actually write
Three common sentences get weaker, and one gets stronger.
“Model A beats Model B on agentic tasks.” Weak unless the claim travelled across scaffolds. A single-scaffold leaderboard is a measurement of A-in-that-harness against B-in-that-harness, and the paper's finding is that the harness term is not small and not constant. This is the sentence in most model launch posts.
“We added 400 tasks, so the ranking is now solid.” Weak in a specific, checkable way. Ask what the 400 tasks varied. If they came from the same generator with the same tooling and the same step limit, they bought precision on the term that was already fine.
“Our agent is state of the art.” Unchanged, and this is the one that survives. A vendor shipping a complete system is making exactly the system-level claim the data supports at 0.935 and up. The irony is worth naming: the leaderboard is more trustworthy for the commercial claim than for the scientific one.
“This scaffold improves any model.” Now has a named measurement to meet. Inter-scaffold reliability is symmetric — if scaffolds do not preserve model rankings, then models do not preserve scaffold rankings either, and a framework's published lift is a claim about the models it was tested against.
Before quoting an agent benchmark in a decision document, write down two numbers next to the score: how many tasks, and how many distinct scaffolds. If the second is 1, the score is a system measurement and you should write the system's name — harness and all — where you were about to write the model's.
How to spend an eval budget after reading this
The practical reallocation is small and unpopular, because it costs breadth that nobody gets credit for.
Run two scaffolds before you run twice the tasks. The cheapest version is not a second framework; it is the same harness with one structural knob moved — a different step limit, a different retry policy, tools defined at a different granularity. If the ranking flips, you have learned the most important thing available to you at that price, and you have learned it before the decision rather than after the migration.
Pool benchmarks rather than extending one. Three 200-task benchmarks that disagree about what a task looks like are worth more than one 600-task benchmark, for the same reason and on the same budget. The practice here is the one we describe in budget-matched evaluation: fix the spend, then argue about allocation.
And keep the scaffold in the artefact. Most eval tooling records the model string and drops the harness configuration into an untracked file, which makes every historical result unreproducible in exactly the dimension this paper says dominates. The HAL-style answer — one standardised harness, published rollouts — works because the scaffold is pinned and visible, not because it is the right scaffold. More on that infrastructure in HAL and async agent eval, and on reading published scores in the benchmark landscape.
The honest caveats
This is a preprint, and its coefficients are estimates from a Bayesian model fitted to sparse, imbalanced data — which is the paper's own stated contribution and also its exposure, since the imbalance it tolerates is imbalance it must model rather than observe. The 0.148 lower bound is one benchmark's figure, not a typical value, and reading the range as a single number is the mistake the range exists to prevent.
It also does not say leaderboards are useless, and anybody quoting it that way has inverted it. The finding is that these instruments are precise about one thing and imprecise about another, and that the published number does not come labelled with which one you are reading. That is a fixable problem, and the fix is a column, not a retraction.
FAQ
What is a generalizability coefficient?
A reliability measure from psychometrics, written Eρ². It estimates how much of an observed ranking would be reproduced if the measurement were repeated under equivalent conditions. Near 1.0 the ranking is dependable; near 0.15 it is close to arbitrary.
What is a scaffold in this context?
Everything around the model that turns it into an agent: the system prompt, the tool definitions and their granularity, the retry and error-handling policy, the step limit, the memory strategy, the harness that runs the loop. Two teams running the same model with different scaffolds are running different systems.
Does this mean agent leaderboards are broken?
No. They are reliable for ranking complete systems — 0.935 to 0.994 across these 22 benchmarks. The problem is that readers convert a system ranking into a model ranking without the data supporting that conversion.
Why can't more tasks fix it?
Because extra tasks reduce task-sampling variance, and the term limiting model-ranking reliability is scaffold variance. Where scaffold coverage dominates, the paper reports that even unlimited similarly built tasks raise model-ranking reliability by at most 0.097.
What should I do instead?
Vary the measurement conditions. Run a second scaffold before doubling the task count, and pool diverse benchmarks rather than extending one. Both change what is being sampled rather than how much of the same thing.
Does this contradict the advice to add tasks rather than runs?
It refines it. Adding tasks still beats adding repeat runs for the variance that repeats address. This paper adds a third term the task lever cannot reach, and says that term is often the binding one for model-level claims.
Further reading
On this wiki:
- Eval variance and statistical power — the task-versus-run argument this paper extends.
- HAL and async agent eval — the standardised harness whose rollouts this analysis uses.
- The benchmark landscape — what published agent scores do and do not establish.
- Budget-matched evaluation — fixing the spend before arguing about the method.
- The agent harness — the scaffold, as a concept, and why it is not a detail.
Sources:
- Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard (arXiv:2610.00651)
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- Harbor Adapters and Harbor-Index — the meta-dataset of task-level scores across scaffolds.