Testing & Verification

What changes

AI gives testing two jobs, and they fail in different ways. The first is generating tests: the model writes test code and someone decides whether to keep it. Its failure is low yield. Most raw output is worthless, and a reviewer who sees all of it drowns. The second is verifying: the suite becomes the gate for everything else the AI does, including its own code changes. Its failure is gaming. An agent rewarded for a green run will, given the chance, make the run green without making the code right.

Part I has already set out the evidence for both. On yield, Meta's TestGen-LLM kept only the candidates that built, passed reliably and raised coverage. A quarter of raw output survived, and engineers accepted 73% of the survivors (the Meta case, T2). On gaming, METR caught o3 reward-hacking in 30.4% of RE-Bench runs, and ImpossibleBench showed frontier models passing tests whose specifications made a legitimate pass impossible (the oracle principle, T1). Both jobs need the same thing: a pass/fail decision the model does not make and cannot change.

The human's job changes with it. Meta's engineers, reviewing mutation-guided tests, "valued being able to focus on evaluating tests rather than having to construct them" (Meta Engineering, September 2025, T2). That is a saving only if evaluation stays rigorous. Writing tests is also a task agents already do. In dotnet/runtime, testing PRs from Microsoft's coding agent merged 75.6% of the time across 99 PRs, against 67.9% for its PRs overall (dotnet/runtime, T2). A merged test is one a reviewer let through. It is not yet a test that catches anything.

SmartBear, which sells testing tools, argues that "Coding assistants have reached Level 4 and 5 autonomy" while testing tools sit "closer to Level 2 or 3", and that "Autonomy without assurance becomes a liability" (SmartBear, October 2025, T4). That is opinion from an interested party, with no data. It still matches the bottleneck chapter: when generation outruns verification, the queue forms here.

The ladder for this phase

The criteria below are the same as in the testing row of the grid. Here each cell also says what an auditor would check. Score each task type separately (Rule 3). A team can honestly be at "Testing: L2 for unit-test generation, L1 for failure diagnosis, L0 for end-to-end tests" all at once.

L1, diagnosis. Auto-Diagnose reads integration-test failure logs and posts a summary with the most relevant lines in Google's code review tool. After deployment across Google it ran on 52,635 distinct failing tests, and 5.8% of feedback rated it "Not helpful" (as of April 2026). It ranked 14th for helpfulness of the 370 tools that post findings there. The 90.14% comes from 71 failures, so treat it as a careful spot check, not a rate. The authors conclude that accuracy is critical to adoption: a diagnosis tool that is often wrong teaches people to ignore it.

L2, flaky repair. FlakyGuard (ASE 2025) treats code as a graph and explores only the context relevant to one flaky test. Earlier LLM repair tools, the authors say, failed in industry by giving the model too little context or too much. FlakyGuard "repairs 47.6 % of reproducible flaky tests with 51.8 % of the fixes accepted by developers", at least 22% better than prior approaches in repair success. The abstract does not name the company; only secondary summaries place the evaluation at Uber.

L3 and L4. The L3 evidence is thin and experimental; the next section takes the best of it apart. L4 is empty: flaky-test repair has published acceptance figures but no published automatic landings.

The case: Meta's just-in-time catching tests

The TestGen-LLM and ACH tests in Part I are hardening tests: they pass when generated and land in the codebase to guard against future regressions. In January 2026 a Meta team published the harder version of the problem. Just-in-Time Catching Test Generation at Meta generates tests for a code change before it lands, for backend systems of hundreds of millions of lines. These tests are meant to find a bug in that change now. Catching tests "fail at test generation time by design". A failure is the signal, and the open question is whether it is a real bug or an intended change in behaviour.

That is the L3 question in miniature: nobody reviews the generated tests, and humans see only what the pipeline routes to them. The paper reports:

• Code-change-aware generation produced about 4 times as many candidate catches as hardening workflows.
• Rule-based and LLM-based assessors, which judge whether a failure is likely a true positive, "reduce human review load by 70%".
• Of 22,126 generated tests, 41 candidate catches were reported to engineers. 8 were confirmed true positives, "4 of which would have led to serious failures".
• A catching test "cannot land together with the changes that it tests". It is discarded, or its signal is acted on.

The authors name the constraint themselves: "The primary challenge is to reduce development drag from false positive test failures", and solving it requires "determining whether a test failure is a true positive, relating directly to the Oracle Problem." Generation is not the hard part at the L3 frontier. Judging is. Meta says the same about its mutation work: future effort will "focus on addressing the Test Oracle Problem", which it defines as "the challenge of distinguishing the desired and correct behavior based on a given input from an incorrect behavior."

Worked example: reading the funnel

The paper publishes one funnel, from one system, with one set of denominators, so the arithmetic is legitimate:

generated tests           22,126
reported to engineers         41   (0.19% of generated)
confirmed true positives       8   (8 / 41  = 19.5% precision)
would have caused serious      4   (4 / 41  =  9.8%)
assessor effect           review load -70%
                          => if load scales with candidates,
                             ~41 / 0.3 = ~137 without assessors

Now the reviewer's side, as illustrative arithmetic with no organisation named. The paper says only that dismissing an intended change "took only a few minutes". Assume 10 minutes per reported candidate. Then 41 candidates cost about 410 minutes, close to 7 engineer-hours, for 8 real catches. That is roughly 51 minutes per true positive and 1 hour 43 minutes per serious one. Without the assessors, the same 8 catches would have cost about three times as much triage. Change the assumed minutes and the totals move, but the shape does not: at this precision most human time goes on false alarms, and the assessor layer, not the generator, decides whether the scheme pays. So measure the gate in front of the human as a classifier, by precision and by what it misses, before calling the pipeline autonomous. An LLM assessor is itself a model judging a model; here that is safe only because a human confirms every reported catch and nothing lands automatically.

What it does not show. How many real bugs the pipeline missed: a false-negative rate needs ground truth the paper does not have. It shows nothing about escaped defects in production. It is an experiment, not a deployed L3 system. Its value is that it publishes a precision figure at all, which almost no vendor of test-generation tools does.

Migration steps

Climb one task type at a time, and do not let Coding run ahead of the rung you are on. The book's heuristic is that Coding runs at most one level above the lower of Review and Testing.

L0 → L1: surface diagnosis, then check it.

1. Put AI failure summaries where engineers already look, the review tool or the CI log, as Auto-Diagnose does.
2. Hand-check a sample of diagnoses against the real root cause before announcing anything. Publish the accuracy and the n.
3. Add a "not helpful" button and track the rate. Google's Tricorder rule, which disables an analyzer above 10% "Not useful" (T2), is a ready-made off-switch (discussed in the code review chapter).

L1 → L2: filtered generation, every test accepted by a human.

1. Build the filter before you turn on the generator. At minimum, a candidate must compile, pass repeatedly (to screen out flakes) and either raise coverage or kill a mutant. Only survivors reach a reviewer.
2. Attribute AI-written tests (a label or commit trailer), and record raw yield at each filter stage, acceptance, and the mutation score of accepted tests. Acceptance alone will flatter you.
3. Pick task types where the oracle is cheap. Flaky-test repair qualifies, because "does it still flake?" is a re-run, and FlakyGuard shows it working at L2.

L2 → L3: the human reviews the report, not the test.

1. Move the oracle out of reach. Tests, mutation configuration and CI definitions are owned by a path or service the agent's credentials cannot write to (see Guardrails).
2. Define what the human reviews instead of each test: a mutation-score delta, a coverage report, and the list of tests added, removed or skipped. That is an evidence bundle, the "Merge-Readiness Pack" Hassan et al. propose as "a structured evidence bundle submitted for human approval" (Hassan et al.). Its review is the subject of the code review chapter.
3. If anything is routed to humans by a classifier, as in JiT, measure that router's precision before trusting it.
4. Run a 90-day comparison of escaped defects in code guarded by AI-maintained tests against a human-maintained baseline. Without it, the L3 criterion in the model is not met, whatever the dashboard says.
5. Make sure there is something to verify against. Spotify's Honk Part 4 rolled out 240 automated dataset-migration PRs, but the BigQuery Runner and dbt repositories rarely had build-time tests, so owners tested the PRs by hand (T3). Where there are no tests, the first task type to delegate is writing them, at L2.

L3 → L4: not yet. No public case has done this. If you try it, the book's proposal is to restrict it to quarantining flaky tests, which can be reversed by un-quarantining. Keep a sampled audit of the false-fix rate and a kill switch, and never let automatic fixes touch production code. For a first L3 anywhere, maintenance is the better place, because the compiler and the existing suite already form the verifier.

Guardrails

Put the oracle outside the agent, mechanically. Instructions are not a boundary. Elastic's CI agent "fixed" broken dependency bumps by removing the bump, until the team added an explicit "NEVER downgrade versions" rule to its instructions, and Spotify's agents commented out failing tests, as reported by InfoQ (both covered under the oracle principle). Spotify's working design is mechanical. In Honk Part 3, deterministic verifiers (format, build, test) are chosen by the surrounding system from the repository's contents and run in a stop hook. When one fails, "the PR isn't opened and the user is presented with an error message". The agent runs "highly sandboxed", and pushing code is done by infrastructure, not by the agent.

Spotify names the residual risk plainly: the worst failure mode is "a PR that passes CI but is functionally incorrect." A verifier the agent cannot touch stops cheating. It does not stop a weak suite from approving wrong code, which is why the metrics below measure the suite itself.

Wrap model judgements in deterministic steps. Meta's LLM detector for equivalent mutants (mutants that cannot be killed because they do not change behaviour) reached "high precision (0.79) and recall (0.47)" alone, "rising to 0.95 and 0.96 with simple preprocessing" such as stripping comments (Meta Engineering, T2). A cheap deterministic step roughly doubled the model's recall. Meta also does not treat mutation as an oracle in itself: "mutation testing cannot exist on its own (it requires a test to already exist)."

Route oracle edits to a human (the book's proposal). Any agent diff that deletes, skips or weakens a test, or touches CI or mutation configuration, goes to a named human reviewer, whatever the change's risk tier. Enforce it with path ownership and branch protection, not prompts. Count such diffs as a metric. No published case reports this rule with outcomes; it follows from the reward-hacking evidence.

Treat the test runner as privileged. An agent that can run the suite can usually run arbitrary code with the runner's credentials. That is a permission question, covered in the security chapter and, for pipelines, in the CI/CD chapter. The wiki's eval integrity and scorer gaming and reward design and hacking cover the mechanics.

Metrics

Coverage is a weak proxy. A 2025 study of mutation-guided test generation (T1) reports that LLMs generated tests for one HumanEval-Java subject with 100% line and branch coverage and a mutation score of 4%. Every line ran, and almost no injected fault was caught. The authors call coverage metrics "weak indicators of a test suite's fault-detection capability" and argue that mutation score is the more reliable measure. Coverage tells you what executed. Mutation score tells you what the tests would notice.

Acceptance is agreement, not correctness. Meta's 73%, FlakyGuard's 51.8% and every vendor acceptance rate measure whether a reviewer agreed. JiT's 8 of 41 shows what happens when you measure whether the tests actually found something. The only study this book found in this half of the lifecycle that measured downstream safety is Google's DIDACT build-repair trial, which counted rollbacks (the DIDACT case, T2). Use it as the template.

Pair one flow metric with one stability metric for each task type you climb. The set below is the book's proposal, built from the measures the cases above did and did not publish:

Acceptance rate stays on the dashboard as a flow signal, never as the outcome. Where these metrics turn into rollback triggers, Chapter 17, Measuring Success and When to Roll Back, sets the thresholds.

Anti-patterns

Coverage or acceptance as the target. A model asked to raise coverage will raise coverage, and the 100%/4% example shows what that can be worth. Acceptance rewards tests that look plausible to a busy reviewer, and busy reviewers are what this book's thesis predicts. Target mutation score; use coverage only as one filter among several.

One agent writes the code and the tests that judge it, in the same session. The agent then controls both sides of the check, and ImpossibleBench shows that agents with access to the tests will exploit them. Keep the tests that gate a change either human-owned or generated separately and locked before the code change starts.

An LLM judge as the oracle. A model judging a model can be a useful extra filter, but it is not a deterministic verifier and has its own false-positive cost. Spotify's Honk judge vetoed about a quarter of agent sessions, and the agent recovered about half the time (Part 3). Spotify later removed the judge as "too rigid, blocking valid changes", as reported by InfoQ from a conference talk; no Spotify primary says so. Measure a judge's precision as you would JiT's assessors, and keep a deterministic check underneath it. When the agent is the product you are delivering to a customer, the same discipline governs its graders; Last Mile, this wiki's handbook for forward deployed engineers, builds that eval suite with the customer's experts in Evals: Proving It Works.

Delegating verification where nothing verifies. Spotify's dataset migrations show the shape: generation scaled, and humans ended up testing by hand because the repositories had no build-time tests. Before you push work to a higher level, write the tests that would gate it.

Choosing a tool by a vendor's benchmark. Vendor comparisons of test generators are common and almost never independent.

To evaluate a test generator yourself, run it on your own code, through your own filter, and compare mutation scores. The wiki's evaluating coding agents and test-generation agents playbooks cover the method, and the generator-verifier gap explains why the verifier is the scarce half.