Evals, in plain words

B10
Concepts · Core Building Blocks

Evals, in plain words.

An eval is a measuring instrument, and the first thing you have to measure with it is itself — because a fifty-case set has a margin of error of roughly ±11 points, so the "6-point improvement" a team ships on Monday is usually indistinguishable from running the same code twice. Three cheap checks separate a scoreboard from a decoration: what your noise floor is, whether your scorer agrees with you, and whether the set can fail at all. Do them before you let a number change a decision.

Step 1What an eval is, and what it is for.

Strip away the jargon and an eval is three things in a list, repeated:

  1. An input — a prompt, a document, a task description, a tool-call request.
  2. An expected behaviour — what counts as good: the right answer, the right action, the right shape.
  3. A scoring rule — how you decide, programmatically or by a human, whether what came back matched.

An eval set is many such tuples run through the same system and aggregated into a score. If you can describe success in code you can write an eval; if you cannot describe it at all, you do not yet have a defined task, which is a different problem. The practice is mature enough to have shared tooling: Inspect, the MIT-licensed framework from the UK AI Security Institute, ships with over 200 pre-built evaluations and first-class support for "model graded" scoring, and every commercial observability vendor now has an eval product bolted to its trace store.

But evals get adopted for one reason and then used for another, and the two have different requirements. Debugging — "what is my agent getting wrong?" — wants a handful of cases you have read individually; twenty is plenty and a thousand is worse, because you will never look at them. Measurement — "is version B better than version A?" — wants enough cases that the difference you care about is larger than the noise. Nearly every disappointing eval programme is a debugging-sized set being asked a measurement question.

Three properties make an eval an instrument rather than a number generator, and all three are measurable in an afternoon: its noise floor (how much the score moves when nothing changes), its scorer agreement (how often the scoring rule agrees with your own judgement), and its failure capacity (whether any case in it currently fails). Skip them and you are not running an experiment, you are collecting a statistic.

Step 2The noise floor, and the arithmetic nobody runs.

Scores move on their own. Sampling is stochastic, tool calls hit live systems, and an agent's trajectory branches — which is why bit-identical reruns are not available. So the first number to establish is not your score but your spread: run the unchanged system three to five times and record the range. Anything inside that range is not a result.

Even with a perfectly deterministic system there is a second, larger source of uncertainty: your set is a sample. For a pass rate p on n cases, the 95% interval is about ±1.96 × √(p(1−p)/n):

cases   observed pass rate   95% interval   smallest change you can see
-----   ------------------   ------------   ---------------------------
   20                  80%        ±17.5 pt   nothing short of a disaster
   50                  80%        ±11.1 pt   large regressions only
  100                  80%         ±7.8 pt   ~10 pt effects
  400                  80%         ±3.9 pt   ~5 pt effects
2,000                  80%         ±1.8 pt   ~2 pt effects

This is why the standard advice to start with 10–100 cases is good guidance badly labelled. Those sets are for finding bugs. If you ship a prompt change, watch the score go from 78% to 84% on fifty cases and declare a win, the honest reading is that you learned nothing: the interval on each measurement is wider than the gap between them.

The fix is not always "collect two thousand cases" — it is usually to change the comparison. Run both versions on the same cases and count only the cases where they disagree. That paired design throws away the cases both got right, which carry no information about the difference, and it is dramatically more sensitive. The test is exact and you can do it on paper:

# 50 shared cases. A and B agree on 44 of them; 6 flipped.
B fixed what A broke  : 5
B broke what A fixed  : 1
# Two-sided exact binomial on 6 flips, 5-1 split:  p = 0.22   -> not significant

# Same set, a change that actually did something: 12 flips, 11-1 split.
# p = 0.006  -> real, on fifty cases, with no new data collected.

Two consequences worth internalising. A change producing six flips on fifty cases is unmeasured, not "a small win" — and the same arithmetic applies to vendor claims and public leaderboards, where a 200-question benchmark reporting 48.6% carries roughly ±6.9 points. Second, read the flips, not the totals: the single case B broke is often more informative than the five it fixed, because it is a regression you now know about. The full treatment, including variance from multi-step trajectories, is eval variance & statistical power.

Step 3The scorer is usually the weakest component.

Scoring rules come in three tiers, and the gap between them is larger than most teams assume.

  • Code-checkable. Exact match, regex, schema validation, a tool-call assertion, "did the invoice total equal the sum of lines". Free to run, deterministic, no drift. Take this whenever the task admits it, and reshape the task so it does more often than you think — asking for structured output instead of prose converts a judgement call into an assertion.
  • Model-graded. An LLM judge with a rubric. Necessary for open-ended output, and the component most likely to be wrong, because its errors are correlated: a judge that rewards confident phrasing rewards it on every case at once, so the bias does not average out with more cases. It also moves under you when the judge model is upgraded — which silently recalibrates every historical score, so pin the version and treat a judge change as a migration.
  • Human. The most accurate and the one you can least afford per run; cap it to a sample and use it to calibrate the tiers above rather than as the routine gate. The honest cost accounting is in the cost of evaluation.

If you use a judge, meta-evaluate it, and the procedure is one afternoon's work: label 50 outputs yourself, run the judge on the same 50, and report agreement — not the judge's accuracy against your labels as though yours were ground truth, but the rate at which the two of you land in the same place, plus a list of every disagreement. The disagreements are the deliverable. In practice they cluster: judges over-reward length and fluency, under-penalise confident errors, and drift when the rubric has more than about three criteria. Until you have run this, you do not know whether your scoreboard is measuring your agent or your rubric. See judge calibration & meta-evaluation, and scorer gaming for what happens when the system under test starts optimising against the scorer rather than the task.

Step 4Build a set that can fail, then put it where it bites.

A set is specific to your task, built from your traffic and your edge cases, and small enough that you can justify every case in it. Two cases are non-negotiable: one trivial case that should always pass — when it fails, your harness is broken, not your model — and one known-bad case you have fixed before, kept as a regression target. Beyond those, the property that decides whether the set is useful is its current score. A set your system already passes at 98% can only detect regressions; it cannot show you an improvement, because there is no headroom left to measure one in. If you want the set to guide development, it has to contain cases you currently fail — and when it saturates, retire cases and add harder ones rather than celebrating.

On public benchmarks: they are for shortlisting, never for deciding. They measure proxy tasks, they saturate into noise once every frontier model clears 95%, they leak into training data (contamination), and vendor tables select the prompts, the harness and the comparison. The full argument is reading benchmarks critically; the short version is that the only number entitled to change your decision is the one your own set produced.

Finally, one distinction to carry out of here. Offline evals run against a curated set in a CI-like loop: you control the inputs, you know what good looks like, and they gate the deploy. Online evals run against production traffic, where you do not pre-know the right answer, so you measure proxies — resolution rate, override rate, thumbs, downstream conversion. You need both: offline catches regressions before they ship, online tells you whether the thing works for real users, and the pipeline that turns online failures into new offline cases is the part that compounds. Read online vs offline evals, then why evaluating agents is hard for the agent-specific problems (error compounding, path dependence, no single gold answer), then eval-driven agent development for wiring it into CI.

Before your next eval-driven decision, spend an hour: run the unchanged system five times and write down the range; compare versions on shared cases and count the flips, not the averages; hand-label fifty outputs against your judge and read the disagreements. If the change you are about to ship produces fewer flips than your noise floor, you have not measured an improvement — and a set nobody currently fails is a regression detector wearing a progress bar.