The Cost of Evaluation

Eval spend scales with change rate, not with traffic.

Every other line on your model bill is variable cost: more users, more tokens, more money, and the ratio holds. Evaluation does not behave that way, and budgeting it as though it does is the root error.

• The driver is change events. A model version bump, a prompt edit, a new tool, a retrieval-config change, a framework upgrade, a provider-side silent update. Each one is a reason to re-run the suite, and none of them correlate with how many customers you have.
• A pre-launch agent has near-zero traffic and its highest eval bill. That is the period of maximum change. Any budget model that allocates evaluation as a percentage of inference spend will underfund exactly the phase where evaluation decides whether you ship.
• It behaves like R&D, so account for it like R&D. Put it in the change budget beside engineering time, not in COGS beside serving cost. The unit economics of a task should not carry an eval amortisation; the decision to change something should carry the cost of proving the change was safe.
• Which is why it gets cut first, and why that is the expensive kind of saving. An eval line item sitting in COGS looks like waste to anyone reading a margin report, and the quarter it gets trimmed is the quarter your change rate is highest. This is the mechanism behind the "eval bill" entry in where the economics breaks, stated as a budgeting error rather than a surprise.

The quadratic that sets the price.

Suppose your agent succeeds on 90% of tasks and you want to know, with conventional confidence, whether a change moved that number. The sample size you need depends almost entirely on how small a move you insist on detecting — and it depends on it quadratically.

# Runs needed to detect a drop from a 90% baseline
# two independent arms, 5% significance, 80% power

  90% -> 80%   (10 points)     ~200 runs per arm      ~400 total
  90% -> 85%   ( 5 points)     ~700 runs per arm    ~1,400 total
  90% -> 88%   ( 2 points)   ~3,850 runs per arm    ~7,700 total
  90% -> 89%   ( 1 point )  ~15,000 runs per arm   ~30,000 total

• Multiply by your cost per run, not per token. A realistic agent task — multi-turn, several tool calls, a long trajectory — commonly lands somewhere between $0.10 and $1.00 of model spend per attempt. At the middle of that range, catching a two-point regression once costs roughly two thousand dollars in inference alone.
• Then add the judge. If grading is done by an LLM, every run is scored at least once more, and a well-built judge often scores twice or uses a stronger model than the agent under test. Judge cost is frequently 20–50% of the total and is the line people forget when they size a suite.
• Then add the labels. Someone had to decide what correct looks like on a few hundred of those tasks, and someone has to re-check that the judge still agrees with them — the ongoing obligation in LLM-as-judge for agents. Human labelling is the capital cost of the whole apparatus and it does not go away.
• The number that shocks people is the per-change one. That entire bill is what one comparison costs. If you make thirty meaningful changes a quarter and re-run at full power each time, you have signed up for thirty of them.

Pair your comparisons before you buy more runs.

The table above assumes two independent samples. Almost no eval needs to be that wasteful, because you control both arms and can run them on the same tasks.

• Run both variants on the identical task set and compare per item. Most of the variance in an agent eval is between tasks, not between variants — some tasks are just harder. Holding the tasks fixed removes that variance from the comparison entirely, and what remains is the difference you actually care about.
• The saving is large and it is free. A paired design tests only the items where the two arms disagree, so the effective sample is the disagreement count rather than the run count. When arms agree on most tasks — the normal case for a small change — this routinely cuts the required runs by several-fold for the same detection power.
• Fix everything you can fix. Same seeds, same task order, same retrieval snapshot, same tool mocks. Every source of run-to-run noise you eliminate is sample size you do not have to buy — the argument developed at length in eval variance and statistical power.
• Report the confidence interval, always. A suite of 60 tasks that shows 88% versus 91% has told you nothing, and the danger is not that people misread it — it is that they act on it, spend a week on a change that was noise, and lose faith in the eval when the "improvement" fails to reappear.

Tier the suite so the expensive one runs rarely.

One suite that runs on every commit is either too weak to detect anything or too expensive to run on every commit. Split it by what the run is for.

• Smoke, per commit, tens of runs. Not a quality measurement — a catastrophe detector. It catches the broken tool schema, the prompt that now returns empty, the auth failure. Cheap enough to ignore the cost, and it catches most of what actually breaks in practice.
• Regression, per release candidate, powered for the effect size you will act on. This is the bill from step two, paid a handful of times a month rather than a hundred. Gate merges on the smoke suite; gate deploys on this one, per eval-driven development and CI.
• Deep, per quarter or per model migration, everything including the slow slices. Adversarial cases, long-horizon tasks, the multilingual slice, the trajectory grading. Expensive, rare, and the run you must not skip when the provider deprecates a model — see model deprecation and migration.
• Route the graders down-tier too. A small local model is adequate for schema conformance, refusal detection, format checks and most objective assertions. Reserve the frontier judge for the subjective slices where it earns the difference — the split argued in small and local models.
• Use the cheap infrastructure you already pay for. Eval runs are batchable and latency-insensitive by definition, which makes them the ideal workload for batch inference discounts, and their shared system prompts make them unusually good candidates for prompt caching. A 50% saving here costs a day of plumbing and applies to every future run.

Know which costs shrink and which are floors.

Optimisation effort is worth spending only on the first list. Teams routinely spend it on the second and conclude that evaluation is unaffordable.

• Shrinks with engineering: tokens per run (caching, shorter trajectories, cheaper judges), infrastructure (batch tiers, self-hosted graders), and re-run frequency (tiering, change batching).
• Shrinks only with a decision: sample size. It is set by your detection threshold, and no amount of cleverness moves it while the threshold stands.
• Does not shrink at all: the labelled golden set and its maintenance. It is a slowly-depreciating asset — tasks go stale, product behaviour changes, and a set nobody has touched in a year is measuring an agent you no longer ship.
• Grows silently: judge drift. When the judge model is upgraded, your historical scores are no longer comparable to your new ones, and re-calibrating against the human labels is a real recurring cost that never appears in anyone's plan.
• Attribute it like any other spend. Tag eval runs by suite, by triggering change, and by team, so the number is defensible when someone asks what it bought. The mechanics are the same as in cost attribution and budgets, and untagged eval spend is the most commonly challenged line in an agent budget.

Decide the threshold first, then buy exactly that much.

The whole discipline reduces to one question asked in the right order, and most teams ask it backwards — they build a suite, run it, and then wonder what its numbers mean.

• Ask: what quality drop would make us roll back? If the honest answer is "five points", you never needed to pay for two-point resolution, and you have just cut your regression bill by a factor of five.
• Ask it per surface, not once. A one-point drop in a safety-relevant refusal rate may be worth thirty thousand runs. A one-point drop in a summarisation quality score is not. Uniform power across a suite means overpaying on most of it and underpaying where it counts.
• Let production do the fine-grained detection. Offline evals are for the changes you have not shipped yet. Small persistent drifts are cheaper to catch in live traffic, where the sample arrives for free — that is the trade in online vs offline evals, and the sizing arithmetic behind it is in detecting quality regressions.
• Batch your changes deliberately. Three changes evaluated together cost one comparison instead of three. The cost is attribution — if the batch regresses you do not know which change did it — so batch changes that are cheap to unpick and evaluate expensive-to-unpick ones alone.
• Publish the cost per comparison next to the result. "This gate costs $1,900 and detects a five-point drop" is a sentence an executive can act on. "We need more eval budget" is not.

Related: why agent evaluation is hard for what you are measuring, eval variance and statistical power for the statistics underneath this page, measuring agent ROI for putting the eval bill on the right side of the ledger, and evals 101 for the ground floor.