Evals: Proving It Works
The eval set is the contract: build it with the customer's experts before the agent, and expect the climb from 80% to the threshold to outlast everything before it. A pass on a test the customer owns is their finding, not your claim. Plan the pilot, the contract and the sponsor's expectations around the climb, or the readout will be judged against a number you never promised.
What this phase is for, and how it fails
Scoping ended with a threshold in the sponsor's words and a held-out set the customer owns. That is a promise. This phase makes it runnable: cases, a grader for each, and a number that comes out the same whoever runs it. Then arguments about whether the agent is good enough become arguments about the eval, which you can settle.
The strongest statement of the idea comes from the vendor side. Colin Jarvis, who runs forward deployed engineering at OpenAI, states his team's rule: a piece of model-written code "is like not done until you have a set of evals that verify the efficacy" (Altimeter interview, November 2025; T4, a vendor's doctrine, quoted from the video's auto-generated captions). OpenAI's enterprise guide makes "Start with evals" its first lesson, and describes Morgan Stanley's first three evals: translation, summarisation, and "Human trainers", which compared the model's answers with those of expert advisors (OpenAI, AI in the Enterprise; T2, a vendor describing its own customer).
The evidence is thin. Almost everything public about evals in engagements is T2, vendors on their own work, or T4, practitioners on theirs; no independent study shows which practices predict a pass. What follows is the practice those accounts agree on: worth following, not proven.
Evals fail in an engagement in three ways.
The vendor grades its own work. The FDE writes the cases, picks the labels and reports the score, so every pass is a vendor claim. Scoping set the fix: the customer labels and keeps the held-out set. This chapter is about keeping that true for three months under pressure to bend it.
The metric is generic. The team adopts an off-the-shelf faithfulness score because the tooling has one. Hamel Husain, who teaches eval practice, is blunt about this: generic evaluations "waste time and create false confidence when you use them as quality measures" (Husain, LLM Evals FAQ, September 2026; T4). The customer did not agree to a faithfulness score. They agreed to "at least as accurate as my senior adjusters".
The early number is mistaken for the finish line. The agent reaches 80% in the first weeks, everyone relaxes, and the remaining distance turns out to be most of the work. Section 9.6 is about that climb.
The composite: five cases, then the stall
Week 2 at the insurer. The scope is signed and the replica is still pending. Before writing any routing logic, the FDE sits with two of the senior adjusters for an afternoon and asks each of them for one claim they would route differently from a junior colleague, and why. Five cases come out of it: a multi-vehicle claim, a claim with an attorney letter buried in an email thread, a total loss, a glass-only claim and a claim filed under the wrong policy. Each gets a label and a sentence of reasoning, and the reasoning matters more than the label. The pattern is Jarvis's description of how OpenAI's FDEs start: "a set of five with the experts from the customer".
Week 3. Read access lands and model risk approves a read-only shadow-run, so the agent sees live auto claims but its suggestions reach nobody. Every disagreement between the agent and the adjuster who actually handled the claim goes into a failure log. Within days the five cases become dozens, all real. Every fixed failure becomes a regression case, so the suite grows each week. Routing accuracy reaches the low 80s quickly. On the held-out set, the senior adjusters agree with each other in the low-to-mid 90s, and that is the bar.
The week-4 readout, on the paused clock. Time-to-first-action meets its target in the shadow-run. Routing accuracy does not. The FDE presents the gap with the failure log grouped into categories, the fix planned for each, and a date to re-measure on the held-out set. The litigation guardrail has held on every case. The COO signs the annual contract with go-live gated on the threshold, and the credit-on-conversion applies.
The stall. For several weeks after the pilot, accuracy barely moves below the threshold. Each fix clears one category and exposes a stranger one. The threshold is reached roughly two to three months after the pilot ended, on the held-out set, measured by the claims director's team. The patterns behind this beat are Jarvis's "five with the experts", LinkedIn's months-long push from 80% toward 95%, and Anthropic's advice to build the first suite from 20 to 50 real failures.
The eval started before the agent, from the people who set the bar, and grew from real failures. The bad news at week 4 came with a plan, so it bought a contract instead of costing one.
Start with five, from their experts
The first eval set is small, and it comes from the customer. Jarvis describes the first step of OpenAI's verification work with a European semiconductor company as "making like a set of five with the experts from the customer" (Altimeter interview, T4). Five is not a sample. It is a conversation that writes down "right" in the experts' words before anyone has a prototype to defend.
Ask for cases that separate an expert from a novice. "Give me an easy one" yields a case the model already gets right. "Give me one a new hire would get wrong, and tell me why" yields the reasoning the agent has to reproduce, and it often surfaces a rule nobody wrote down. In the composite, the attorney letter buried in an email thread is that case: no manual mentions it; every senior adjuster knows it.
Write each case as four lines: the documents, the right queue, the expert's one-sentence reason, and the rule the reason implies. The rule column is the start of your list of checks that belong in code, not in the prompt; the next chapter builds them.
Each case must be one the experts agree on. Anthropic's engineering guide to agent evals puts the test well: "A good task is one where two domain experts would independently reach the same pass/fail verdict" (Anthropic, Demystifying evals for AI agents, January 2026; T4, vendor guidance). If two of the customer's experts disagree on a case, it does not belong in the eval yet. It belongs on a list of questions for the threshold owner, because the disagreement is a policy question, not a model question.
Before the five, there is a cheaper check worth running. Jarvis describes testing whether a use case is feasible at all by trying it by hand, "with an N of 10", and concluding that if it works "seven or eight times, then like maybe this is going to work in production" (T4). That tells you whether to start, not whether you are done. A feasibility check that quietly becomes the acceptance test is how teams ship on ten examples.
Grow it from real failures
Five cases define "right". They do not show how the agent goes wrong; for that you need it running on real inputs, and someone reading what it did.
The sources agree on scale, though all of it is vendor or practitioner advice. Anthropic: "20-50 simple tasks drawn from real failures is a great start" (T4). Eugene Yan, writing on product evals: "I usually recommend having at least 50-100 failures out of 200+ total samples", and he prefers organic failures to invented ones: "I find these synthetic defects problematic" (Yan, Product Evals in Three Simple Steps, November 2025; T4). Husain recommends reviewing at least 100 traces, each labelled pass or fail, because "Binary evaluations force clearer thinking and more consistent labeling" (T4).
In an engagement the real inputs usually arrive in two stages. Before read access, you have the redacted export and sanitised fixtures from the data-access chapter, which are good for building and poor for finding failures: nobody sanitises in the strange cases. Once access lands, run a shadow: the agent processes live work, its output reaches nobody, and each output is compared with what the human actually did. Every disagreement is a candidate failure. The shadow-run is the best eval source you will get, and it needs the review that the security chapter starts on day 1.
Then read the failures and sort them. Teams skip this step, and it is where the time goes. Husain reports that on past projects "we've spent 60-80% of our development time on error analysis and evaluation" (T4). His account of one client, a property-management assistant called Nurture Boss, shows why sorting pays. There, "just three issues accounted for over 60% of all problems", and focused work on one of them, date handling, raised its success rate "from 33% to 95%" (Husain, A Field Guide to Rapidly Improving AI Products, March 2025; T4, one consultant's account of his own client). Without categories every fix is a guess; with them you know which fix moves the number most.
Who labels matters as much as how many. Husain argues for one domain expert, whom he calls a benevolent dictator, who can take others' input but "they drive the process" (T4). In an engagement that is usually the threshold owner or someone they name. A committee of labellers produces a committee's definition of correct, and the threshold owner will not sign it.
Three sets, three jobs, and graders you can trust
By the end of the first month you have three collections of cases. Mixing up their jobs is the commonest way an eval stops meaning anything.
- The held-out acceptance set. Labelled by the customer's experts in scoping, owned by the customer, and used only to measure against the threshold on agreed dates. You never tune on it. If you look at its failures to decide what to fix, it has become a development set, and you need a new one.
- The development set. The five, plus the failures from the shadow-run. This is the set you work against every day. Its score should start low; Anthropic says capability evals "should start at a low pass rate", which gives the team "a hill to climb" (T4). A development set you pass easily is not finding anything.
- The regression suite. Every failure you have fixed, kept so that it stays fixed. Anthropic's bar is that regression evals "should have a nearly 100% pass rate" (T4). It runs on every change: a prompt edit, a retrieval change, a model upgrade.
The regression suite is also what lets an engagement grow. On Klarna's customer-service assistant, Jarvis says, OpenAI's team wrapped each intent in its own set of evals, so it could scale "from like say 20 policies all the way up to 400" (Altimeter interview, T2). Sierra builds the same mechanism into its product, promising to "run the regression test suite for every one of our live customers" on platform upgrades (Sierra, June 2024; T4, a vendor's promise). The suite makes the next scope expansion safe, and it is the asset you leave at handoff.
It is also only as good as what it measures: Klarna later said its AI-first customer service had produced "lower quality", a reversal When Engagements Fail examines, and nothing public says whether the per-intent evals measured resolution or only containment.
That is the diagnosis when the scores pass and the users hate it. Once suggestions reach users, a propose-only agent logs the evidence for you: suggestions overridden or ignored. Pull a few dozen overrides from a week where the eval passed, and give them to the threshold owner's experts to label. If the agent was right, the problem is trust or the queue view, and it belongs to adoption. If the agent was wrong in a way no grader checks, add that dimension as a new grader and new cases, and re-sign the plan. Never argue with an override using an eval score.
Graders. Use the cheapest grader that is right. Exact checks in code come first: did the claim land in the right queue, is the output valid against the schema, was a litigation-flagged claim kept out of the fast queue. A model acting as judge is for what code cannot check, and it is calibrated against the experts' labels before anyone trusts it. Yan's method is to align the judge on part of the labelled data, "hold out the remaining 25% as the test set", and measure agreement with Cohen's kappa, where "anything above 0.7 is excellent" (T4). Build "one evaluator per dimension"; of a single judge asked to score many dimensions at once, he writes that he has "never seen this work well". The wiki's page on judge calibration covers the method.
Consistency. The same case can pass on one run and fail on the next. Anthropic distinguishes pass@k, "the likelihood that an agent gets at least one correct solution in k attempts", from pass^k, "the probability that all k trials succeed", and recommends pass^k "for agents where consistency is essential" (T4). A claims router is one of those: an adjuster who sees the same claim routed two ways will stop trusting both. Run each held-out case several times and report the share that pass every time.
When the number looks wrong, suspect the grader. Anthropic warns that a zero pass rate across many trials "is most often a signal of a broken task, not an incapable agent". It adds: "You won't know if your graders are working well unless you read the transcripts and grades from many trials" (T4); its own example is in the box. In an engagement the commonest broken grader is a wrong label: a held-out case the experts labelled in a hurry. On the development set, read ten failures yourself before you report a category. On the held-out set, do not: hand the failing case IDs to the threshold owner, let their experts recheck the labels, and log every relabel with its reason and date. A relabel the vendor makes is a thumb on the scale; one the customer makes and signs is a correction.
A broken grader, measured. Anthropic reports that one of its models, Opus 4.5, "initially scored 42% on CORE-Bench", partly because of "rigid grading that penalized “96.12” when expecting “96.124991…”". "After fixing bugs and using a less constrained scaffold", the score "jumped to 95%" (Anthropic, January 2026; T2, a vendor reporting on its own model). Model names and benchmark scores date quickly; the point that a broken harness can hide more than half of a real score does not.
The engineering, from suites in CI to keeping them current as inputs drift, is in the wiki: agent evaluation, eval-driven development and CI, and eval-set maintenance. The SDLC book's chapter on testing and verification makes the parallel argument for code that agents write.
The long climb from 80%
The best-documented fact about evals is the one engagement plans most often ignore: the last stretch takes longest. LinkedIn's engineering team reported that "The team achieved 80% of the basic experience we were aiming to provide within the first month" and "then spent an additional four months attempting to surpass 95% completion of our full experience". The early pace, they wrote, "created a false sense of 'almost there'" (LinkedIn Engineering, April 2024; T2, a first-party engineering account). That is one team, not a law, but every account here has the same shape.
Morgan Stanley is the engagement version: by Jarvis's account the technical problem took weeks and the pilots and evals before advisors trusted it took months, a trust period that Production and Handoff takes up (T2).
The counter-case is a climb that never finishes. McDonald's tested voice ordering with IBM in about 100 locations; a 2022 analyst report put accuracy in "the low 80% range" against a "95%-plus" target, and the test ended in 2024 (Nation's Restaurant News, June 2024; T3, the accuracy figure second-hand). When engagements fail reads it as a failure case. Here the point is narrower: nobody set a date by which the climb had to succeed.
Four things follow for an FDE.
Know the ceiling. Scoping measured how often the experts agree with each other; no agent can be shown to beat that. In the composite the seniors agree in the low-to-mid 90s, and "at least as accurate as my senior adjusters" means matching them, not beating them. Write the ceiling into the eval plan next to the threshold, so nobody later quotes a target the experts themselves miss.
Plan the commercial terms around the climb. A 30-day pilot will rarely end above the threshold on a hard problem; pretending otherwise sets up the conversion meeting to fail. The composite's answer is to convert on evidence of the climb, with go-live gated on the threshold. That only works if the sponsor heard about the climb before the readout, not at it.
Count experiments, not features. Husain's rule for this phase is that "The key metric for AI roadmaps isn't features shipped – it's experiments run" (T4). During a stall, a weekly note is the evidence that the climb is happening, even when the headline number did not move. Three lines: "Attacked: multi-vehicle claims, the largest category. Tried: splitting the packet per vehicle before routing. Moved: that category's misses on the development set fell by about half; held-out re-measure on the agreed date."
Write the stall rule before the stall. At the readout, agree what happens if the number stops moving. A workable rule: if two held-out re-measures in a row barely move, and most remaining misses are cases the experts themselves split on, the open question is the threshold's definition, not the agent. It goes to the threshold owner in writing. If the experts agree on the remaining misses, the climb continues, to a last date set now. Without a rule agreed in advance, a stall becomes a negotiation about blame.
Delivering the bad news
At some point you will tell a sponsor the agent is not good enough yet, and how you do it decides whether the engagement continues. There is no study of this; what follows is practice.
Lead with the number against the threshold, in their words. "Routing accuracy is in the low 80s; your senior adjusters agree with each other in the low-to-mid 90s; the bar you set is to match them. We are not there." No rounding up, no "nearly", no switching to a kinder metric. Soften it here and every later number is discounted.
Then show what passed. In the composite, time-to-first-action met its target and the guardrail held on every case. That is not a consolation prize: it shows which parts of the promise are kept, and narrows the problem to one number.
Then the plan. Show the failure categories with the rough share of misses in each, the fix for each, and the order. Give the re-measure date and who measures. Categories make a plan credible because they show you know why the agent fails. "We'll keep improving it" is not a plan.
Then what does not move. The threshold does not change because you missed it. Lowering it is the threshold owner's decision, in writing, and never your suggestion. Changing the bar to fit the result is how a pilot "succeeds" and is then abandoned in production.
Finally, give the sponsor a decision. The useful options are concrete: continue to the re-measure date on the current terms; narrow the scope to the categories that already pass; or stop. Recommend one, and say why. A sponsor handed a decision owns the bad news; one handed only bad news looks for someone to blame.
Tell the threshold owner first, privately. They should never learn the result in front of their boss, and they will know which categories the sponsor will care about.
The template: the eval plan
The eval plan sits next to the one-page scope and makes its metrics runnable. Keep it where the customer can see it, and re-sign it whenever a set, a grader or the threshold changes.
EVAL PLAN version / date: ________ 1. CONTRACT (from the one-page scope) Threshold, precise: ___________________________________ Threshold, sponsor's words: ___________________________________ Ceiling (expert agreement): ___________________________________ Guardrail (never): ___________________________________ Threshold owner: ___________________________________ 2. SETS Source Size Owner Used for held-out _____________ ______ ________ readouts only development _____________ ______ ________ daily work regression fixed failures grows ________ every change Vendor tunes on or reads held-out failures: NO 3. LABELS Labelled by: ________________ Adjudicator: ________________ Label form: pass / fail per dimension, plus one-line reason Relabels: customer only; log case id, reason, date, initials 4. GRADERS (one per dimension) Dimension Grader: code / judge / human Calibrated (kappa) ______________ ____________________________ _________________ ______________ ____________________________ _________________ 5. CONSISTENCY Trials per held-out case (k): ____ Report: pass^k 6. CADENCE Regression suite runs on: every prompt / model / retrieval change Held-out measured on: ____________________ (agreed dates only) Stall rule: if ___ re-measures barely move, then ______________ 7. FAILURE LOG Share of misses Fix Owner ____________ _______________ _______________ ________ ____________ _______________ _______________ ________ 8. READOUT Date: ________ Threshold owner briefed first: ________ Re-measure date: ________ Decision asked for: ______________ 9. SIGN-OFF Threshold owner (cases and labels are right): _______ date: ____ Vendor lead (we will measure as written): _______ date: ____
Filled from the composite at the week-4 readout. Figures are ranges; a real plan carries the measured numbers.
EVAL PLAN: auto FNOL triage version 3 / week 4
1. CONTRACT (from the one-page scope)
Threshold, precise: routing accuracy >= senior-adjuster
agreement on held-out; first action
same work day
Threshold, sponsor's words: "at least as accurate as my senior
adjusters, and nobody waits a day"
Ceiling (expert agreement): low-to-mid 90s
Guardrail (never): litigation-flagged claim to fast queue
Threshold owner: claims director
2. SETS Source Size Owner Used for
held-out week-1 sample a few claims readouts only
hundred ops
development 5 expert cases dozens FDE daily work
+ shadow misses
regression fixed failures grows FDE every change
Vendor tunes on or reads held-out failures: NO
3. LABELS
Labelled by: 2 senior adjusters Adjudicator: claims director
Label form: pass / fail per dimension, plus one-line reason
Relabels: customer only; log case id, reason, date, initials
4. GRADERS (one per dimension)
Dimension Grader: code / judge / human Calibrated (kappa)
right queue code: compare to label n/a
litigation flag code: never in fast queue n/a
reason is sound judge, checked by seniors > 0.7 before use
5. CONSISTENCY
Trials per held-out case (k): several Report: pass^k
6. CADENCE
Regression suite runs on: every prompt / model / retrieval change
Held-out measured on: week 4; then every two weeks to go-live
Stall rule: if 2 re-measures barely move, and most misses are
cases seniors split on, threshold owner restates
the bar in writing
7. FAILURE LOG Share of misses Fix Owner
multi-vehicle largest share split packet FDE
per vehicle
poor scans, a sizeable share OCR pass before FDE
handwriting routing
wrong policy a smaller share policy lookup FDE
check in code
label errors a few relabel, logged seniors
8. READOUT
Date: end of week 4 Threshold owner briefed first: yes
Re-measure date: two weeks on Decision asked for: sign annual
contract, go-live gated on the threshold
9. SIGN-OFF
Threshold owner (cases and labels are right): claims dir. wk 4
Vendor lead (we will measure as written): the lead wk 4
Two things in the filled version are easy to miss. The last failure category is the labels, not the agent, and it is cheaper to find before the readout than during it. And the sign-off asks the threshold owner to vouch for the cases, not the score: the score is what the plan produces, the cases are what the customer agrees to be judged by.
Failure signals
Each of these shows up before the readout, and each points to one fix.
Ready to move on
The production build can start once the first five are ticked. Go-live waits for all six:
- The threshold and the guardrail run as checks, on a held-out set the customer labelled and owns.
- The development set is built from real failures, sorted into categories, and the regression suite runs on every change.
- Every model-based grader has been calibrated against the experts' labels.
- Expert agreement on the held-out set has been measured and shown to the threshold owner.
- The sponsor has heard the current number, the plan and a re-measure date, from you, before any readout.
- The threshold has been met on the held-out set, measured by the customer's team, and signed by the threshold owner.
The composite builds in parallel through the stall. Building in their environment, the next chapter, covers that build.
Let the customer's experts write the first five cases, and let the customer keep the test. An eval you wrote measures your understanding of the problem. An eval they wrote measures the agent against theirs, and only that result settles the argument about whether it is good enough.