Budget-Matched Evaluation

The budget is a dial on the harness, not a property of the model.

Every agentic benchmark result is produced by a model inside a scaffold, and the scaffold decides how much the model is allowed to spend. Four dials do nearly all of the work, and all four belong to the submitter rather than to the weights:

• Step ceiling. How many tool-calling turns before the run is abandoned. Raising it converts timeouts into solves.
• Sampling. How many attempts, and whether a verifier picks among them. This is the largest single multiplier available.
• Thinking effort. On models that expose it, a reasoning budget per turn — the dial in adaptive thinking and effort budgets.
• Retry and repair policy. Whether a failed test re-enters the loop, and how many times.

Turn all four up and almost any competent model's score rises, which is the whole content of inference-time scaling. So a published number is a score for three things — model, harness, budget — and conventional leaderboards name only the first. The agent harness makes the general version of this argument; this essay is about the third term specifically, because it is the one that can be equalised cheaply and almost never is.

Curves cross, so a single point cannot be extrapolated.

If accuracy were a fixed monotone function of budget with the same shape for every model, the mismatch would be an annoyance rather than a defect: you could rank at any budget and get the same order. It is not. Three well-documented behaviours make the curves differently shaped, and any two of them are enough to produce a crossing.

• Early saturation. A smaller model exhausts what it can do in a few steps and then loops, so its curve flattens quickly. Extra budget buys repetition, not progress.
• Late payoff. A model trained to use long trajectories keeps climbing well past the point where its rival has stopped — and is therefore worse at a tight cap and better at a loose one.
• Non-monotonicity. More thinking sometimes lowers accuracy, by talking a correct early answer out of itself or wandering into an unrecoverable state: inverse scaling in test-time compute.

Take the October 2026 pair above at face value. The two models are 0.9 points apart on score and 3.2× apart on tokens per attempt. A reader who needs an agent to finish inside roughly 30,000 tokens cannot tell from those two numbers which model to pick, because the only published point for the second model sits at 80,000 — and nothing in a single point says whether its curve at 30,000 is at 78% or at 40%. The mismatch is not that the comparison is close. It is that the comparison has not been made.

This is also why the response "just divide" fails. Score per thousand tokens is a ratio of two numbers measured at one operating point, and it implies a linear curve through the origin, which no agent has. The arithmetic looks like a normalisation and is an assumption.

Cost-per-solve describes a submission; budget-matching constrains one.

Princeton's HAL was the field's first real move here, and HAL and asynchronous agent eval covers it properly: publish total spend divided by solved instances next to the pass rate, so a procurement team can see that a three-point lead cost four times as much. Keep it. It is not, however, budget-matching, and the difference decides what each instrument can answer.

• Cost-per-solve is descriptive and post-hoc. It measures what this submission chose to spend. Two entries can be reported side by side with budgets that differ by an order of magnitude; the metric records the difference rather than removing it.
• Budget-matching is prescriptive and prior. Every entry gets the same ceiling, and the ceiling is yours. It answers "which of these wins under my constraint", which is the only question a deployment decision asks.
• Dollars drift; tokens and steps do not. A cost-per-solve figure is partly a statement about last quarter's prices, and token prices have moved by large factors every year — price deflation and cost per task. Match on a physical budget and convert to money at the end.

Use both. Match the budget to decide, and report cost-per-solve so the decision can be re-priced next quarter without re-running anything.

Three ways to match, and what each one hides.

They differ in cost and in what they conceal, and the right choice depends on whether you are picking a configuration to ship or a model to live with.

Hard cap

Give every candidate an identical ceiling — tokens, steps, wall-clock — and score a run that hits the ceiling as a failure. Cheapest to run, and the most faithful to production, because production has a ceiling too and a truncated run is a failed task to the person waiting for it. What it hides is proximity: a model one step from success scores the same as one that was lost at step two. That is the correct thing to hide if your cap is real, and the wrong thing if you were only guessing at it.

Equal spend

Give every candidate the same total budget and let it decide how to spend it — more samples, or longer trajectories, or more thinking per turn. This measures allocation skill as well as raw capability, and allocation skill is a real production property. What it hides is the shape of the spend: eight parallel attempts and one long attempt cost the same and feel completely different to a user, which is why it needs pairing with measuring agent latency.

Frontier reporting

Score every candidate at three or more caps and publish the curve rather than a point. Three times the cost and the only method that reveals a crossing — which means it is the only one that can tell you your cap is in the wrong place. Worth it for a model you will build a product on; not worth it for a weekly regression run.

# Budget-matched run, three caps. Truncation counts as failure.
CAPS_TOKENS = [20_000, 50_000, 120_000]

for model in candidates:
    for cap in CAPS_TOKENS:
        runs = harness.run(model, tasks, token_cap=cap, step_cap=40,
                           samples=1, on_exceed="fail")
        report(model, cap,
               pass_rate=runs.pass_rate(),
               p50_tokens=runs.quantile("tokens", 0.50),
               p95_tokens=runs.quantile("tokens", 0.95),
               truncated=runs.share("truncated"))

# Read the truncated share first. Above ~15% the cap is the ceiling,
# not the model, and the pass rates at that cap compare harnesses.

The last line is the one teams skip. If a large share of runs is hitting the ceiling, you are measuring the cap; move it until truncation is a small minority, then compare.

pass@k is a budget, and it is the commonest mismatch in the wild.

A pass@8 number is eight attempts' worth of spend. Comparing it to someone else's pass@1 is a budget mismatch wearing the costume of a metric choice, and it is the mismatch most likely to be in a slide you are shown this quarter. Two corollaries follow.

• pass@k requires a picker you actually have. The metric credits a solve if any attempt succeeded, which assumes something downstream can tell which one did. If your production path has no verifier, pass@k is not an estimate of your production accuracy; it is an upper bound you cannot reach. Where a verifier does exist, that is the architecture in verifier-guided search.
• pass^k is the reliability question, not the budget question. The probability that all k attempts succeed is what matters when a human is not checking each one. It is also noisier, which is the subject of eval variance and statistical power — and budget-matching does not rescue you from variance. A matched-budget comparison on too few tasks is a precise measurement of noise.

Best-of-n with a reranker, debate, self-consistency and reflection loops are all the same thing in this frame: multipliers on spend that belong inside the budget you matched, not outside it. If one candidate's configuration includes a reflection pass and the other's does not, you are comparing harnesses — honest only if you say so.

Pick the cap from your product, then measure at it.

The cap is not an evaluation parameter to be tuned until the numbers look good. It is a fact about the thing you are building, and it is derivable before any model is chosen.

• Derive it from two constraints. Your latency budget gives a wall-clock ceiling; your cost-per-successful-task ceiling, divided by your token prices, gives a token ceiling — the arithmetic in unit economics. Take the tighter of the two and write it down before you look at a leaderboard.
• Instrument three budgets separately. Tokens, steps and wall-clock bind at different moments, and an agent that is cheap and slow fails differently from one that is fast and expensive — timeouts and deadline budgets.
• Keep one uncapped arm as a diagnostic. Run a small slice with the ceiling removed. The gap between capped and uncapped pass rate tells you which thing is limiting you: a large gap means your cap is the ceiling and the engineering job is to make the loop cheaper, while a small gap means the model is the ceiling and no budget will fix it. Most teams never learn which situation they are in.
• Refuse the bare score in review. "84% on the benchmark" is not a reportable result. "84% at a 50,000-token cap, 6% truncated, p95 at 41,000" is. Make the pair the house style, the way reading benchmarks makes provenance the house style.

Related: human baselines in agent evals for the denominator a score needs, the 2026 benchmark landscape for which tests still rank anything, and fewer tokens vs cheaper tokens for what a budget win is worth once it reaches the invoice.