Maintaining an eval set.
Your eval set loses value every time it does its job, because every regression you fix against it is a case that will never separate two candidates again — and a suite where all your candidates score the same has stopped being a measurement while continuing to cost money and produce a number people trust. Treat the set as a depreciating asset with a replacement budget, and the first thing to measure is not your agent but the fraction of cases everything already passes.
Saturation is caused by your own success, not by neglect.
The lifecycle is mechanical. You build a set from the failures you know about. You ship fixes until they pass. Each fix converts a discriminating case into a permanent pass — right where the interesting behaviour used to be. Six months later the suite is 300 cases, the pass rate is 94%, and roughly 280 of those cases would have returned an identical verdict for any candidate you might plausibly ship. You are paying to run 300 and learning from 20.
This is not decay through rot; it is decay through fitting. It is the same phenomenon as a benchmark going stale in public, arriving privately and much faster, because your team is optimising against this specific set with far more iterations than any lab gets against MMLU. The suite is now a regression harness — genuinely valuable for catching things you already fixed coming back — but it is no longer a discriminator, and teams keep reading it as if it were.
The number that tells you where you are:
saturation = cases passed by ALL candidates in the last N comparisons
---------------------------------------------------------
total cases
# candidates = model versions, prompt versions, harness changes
# you actually compared over the last quarter
Above roughly 0.8, most of your eval bill is buying confirmation. Below 0.5 on a mature system, your set is probably too hard to be a release gate and should be split.
Score the cases, not just the system.
Every case in the suite has a property nobody records: how often it has changed anyone's mind. Store it. For each comparison you run — a model upgrade, a prompt change, a tool-catalog edit — write down per case whether the two candidates disagreed. After a quarter you have a per-case discrimination count, and the distribution is always more lopsided than people expect: a small minority of cases carry nearly all the signal, a long middle occasionally fires, and a large tail has never once separated anything.
Two things follow directly:
- You can shrink the CI gate without losing coverage. Run the high-discrimination cases on every commit and the full suite nightly or per release. That is usually a 5–10× cost reduction on the loop developers actually wait for, which matters because eval spend scales with change rate rather than traffic — the arithmetic is in the cost of evaluation.
- You can see which capability you have stopped testing. Group discrimination counts by the tag on each case (tool selection, recovery from a bad result, multi-step planning, refusal handling). A category whose cases all stopped firing is either solved or untested, and those two look identical on a dashboard.
A case that has never separated two candidates is not automatically worthless — some exist to catch a specific catastrophic regression that has correctly never recurred. Tag those explicitly as tripwire and exempt them. The point of the count is to stop the untagged majority from silently becoming ballast.
Give the set a replacement rate and hold to it.
Eval sets grow monotonically because adding is somebody's job and removing is nobody's. Fix that with a standing rate rather than a periodic cleanup: some fixed share of the suite is replaced each cycle — a reasonable starting point is 10–15% per quarter, tuned so that the median case age is under a year on a system you change weekly.
Retirement needs a rule so it does not become an argument. A case leaves the suite when it has run in at least N comparisons, discriminated in none of them, is not tagged as a tripwire, and its capability tag is still represented by other live cases. Archive rather than delete: keep it, keep the label, and re-run the archive once a year, because "solved" is a claim with a shelf life — a model swap or a context-policy change can revive an entire retired category at once.
Budget the labelling that replacement implies. New cases need ground truth, and ground truth costs annotator time that competes with everything else; if two of your experts only agree on 72% of traces, a new case built on one expert's opinion is noise you are about to gate releases on. That constraint is the subject of annotation and labeling ops, and it is the real ceiling on how fast a set can be refreshed.
Source replacements from production — and correct for what production cannot show you.
The best new cases come from real traffic, and the pipeline is worth building once: sample traces, filter for the interesting ones, label, minimise into a reproducible case with a fixed environment and a checkable outcome. The production feedback signals page covers which signals are worth mining; trace sampling and retention covers keeping enough trace to reconstruct one.
The trap is that this pipeline inherits a bias you have to correct by hand. You can only harvest failures you noticed, so the set fills up with loud failures — errors, timeouts, thumbs-down, escalations — and systematically under-represents the confident wrong answer that nobody flagged, which is the failure mode that actually costs you. Three counterweights:
- Reserve a slice of the replacement budget for adversarial and synthetic cases aimed at the silent failures: fabricated tool results, contradictory instructions, injected content, a task that should have ended in a refusal or an abstention.
- Sample some traces at random, not only the flagged ones, and label them blind. The disagreement rate between "flagged" and "randomly sampled" tells you how big your detection blind spot is.
- Preserve the natural difficulty mix. A set harvested purely from failures drifts toward pathological inputs and stops predicting anything about the median request — see why agent eval is hard.
Keep a holdout you never debug against, and assume the rest leaks.
Split the suite at creation: a development set you may look at, iterate on, and fix against, and a holdout you run at release time and never inspect case by case. The discipline is unpopular and it is the only thing standing between you and a number that measures how hard you tried. The moment an engineer opens a holdout case to work out why it failed, that case has joined the dev set — make that the explicit rule, move it, and replace it.
Assume leakage on top of that. Cases sent to a hosted API are outside your control; team members memorise the ones they debug; a case pasted into an issue thread may be indexed. This is the private version of the problem in benchmark contamination, and the practical consequence is that a holdout has a shelf life measured in quarters, not years. Rotate it on a schedule, keep a small never-run reserve to rotate in, and treat a suspiciously large jump on the holdout the same way you would treat one on a public benchmark: as a contamination hypothesis until you have ruled it out.
Version the set, and re-baseline instead of comparing across versions.
A score means nothing without the suite version attached, and the most common way a real regression ships as an upgrade is a comparison drawn across a set that changed in between. Two rules make this safe:
- Pin and stamp. The suite gets a version, and every reported score carries suite version, judge version and harness version alongside the model — the same tuple discipline as rollout and versioning. A judge prompt edited between two runs is a silent change to the ruler; if you use a model judge, its calibration decays on its own schedule, which is the argument in LLM-as-judge for agents.
- Re-baseline on every set change. When you retire and add cases, re-run the current production candidate on the new suite before anything else. That number, not last quarter's, is the bar. It costs one extra run and it removes the single most expensive class of eval mistake.
Expect the re-baseline to look like a regression, and say so in advance. Replacing solved cases with live failures mechanically lowers the pass rate, and a team that has not been warned will read a healthy refresh as a quality drop and roll something back. Publish the two numbers side by side — old suite and new suite, same candidate — and the conversation resolves in a minute. The same statistical caution applies as everywhere else here: with a judged metric, a few points of movement on a few hundred cases may be noise, and eval variance and statistical power is what tells you which.
Do this in the next week, in this order. One: pull your last quarter's comparison runs and compute, per case, how many times it separated two candidates — then look at the fraction that never did. Two: take the top decile by discrimination and make that your per-commit gate, moving the rest to nightly. Three: pick a retirement rule, archive the dead cases under it, and put the freed budget into replacements harvested from production plus a reserved slice for silent failures. Four: split off a holdout and write down, as policy, that opening a holdout case moves it. If you only do the first step, you will still learn more about your eval suite in an afternoon than the pass rate has told you all year.
Related: eval-driven agent development for how the gates are structured, detecting quality regressions for the production-side counterpart, outcome vs trajectory evaluation for what a case should assert, and evals 101 for the ground floor.