Test-Generation Agents

The oracle problem, and why generated tests inherit your bugs.

A test has two halves: the setup that drives the code, and the oracle that says what the right answer is. Agents are very good at the first half and structurally handicapped on the second, because the only artifact in the repository that describes intended behaviour precisely is the implementation — and the implementation is the thing under test.

• Read the code, encode the code. Given discount(order) that returns 0 for negative totals when it should raise, an agent working from the source writes assert discount(bad) == 0. The test is correct about what the function does and wrong about what it should do, and it now stands between you and the fix. The next engineer to correct the function gets a red suite and, more often than anyone admits, corrects the test instead.
• This is not a capability gap that scales away. No amount of model improvement lets a reader recover intent from a description of behaviour that never stated intent. The fix is upstream: give the agent an oracle that is not the code — a docstring that states a contract, a type signature with real constraints, an issue describing the expected output, a reference implementation, a property that must hold.
• Assertion-free tests are the honest failure. A generated test that calls the function and checks only that it did not throw is at least not lying. It is also worth close to nothing, and it will inflate every coverage number you have. Grep new suites for tests with no assertion or a single truthy assertion; the count is usually higher than expected.
• Mocks make it worse. An agent that cannot make a dependency work will mock it, and a test whose collaborators are all mocked asserts that your mocks behave the way you told them to. It passes forever, including after you delete the real integration.

Coverage is the wrong gate. Mutation is the right one.

Line coverage measures which lines executed while the suite ran. It does not measure which behaviours are pinned, and a suite of assertion-free tests can reach 90% while pinning nothing. Handed a coverage target, an agent will hit it — that is a well-shaped optimisation objective and a badly-shaped proxy for what you wanted.

• Make the gate a failure, not a pass. Mutation testing perturbs the source — flips a comparison, changes a constant, deletes a statement — and re-runs the suite. A mutant that survives is a behaviour nothing tests. Mutation score is the number to put on the pull request, because a generated test that cannot detect an inverted conditional is measurably worthless and coverage will never say so.
• You do not need a full mutation run to get the benefit. Mutation testing is expensive on a large repository, but you are not testing the repository — you are testing the diff. Mutate only the lines the new tests claim to cover and the run collapses to seconds, which is cheap enough to sit in CI as a blocking gate.
• The cheap version works too. Before merging a generated suite, revert the function under test to a deliberately broken version — an early return None, an inverted boundary — and confirm the suite goes red. One scripted check catches most of the tests that assert nothing. If it stays green, the tests are decoration.
• Never let the agent see the gate's internals. If the mutation harness is a tool the agent can call and iterate against, it will overfit to the mutants rather than to the behaviour, which is reward hacking with extra steps. Run the gate after generation, on the artifact, and report only pass or fail.

Where generation actually pays: when the oracle already exists.

The oracle problem is not universal — it is specific to writing new specifications. There are three jobs where the correct answer is already sitting somewhere outside the code, and on those a test-generation agent is genuinely excellent rather than merely productive.

• Characterization tests before a refactor. Here you do not want the tests to be right, you want them to be faithful — a net that captures exactly what the code does today so that a rewrite can be checked against it. The implementation being the oracle is the entire point, the bugs get pinned deliberately, and volume is a virtue. This is the highest-return use of the technique and the enabling step for large-scale migration.
• Reproduction tests from a bug report. The report contains the oracle: this input, that wrong output, this expected one. Turning it into a failing test is mechanical, verifiable (the test must fail before the fix and pass after), and it is the deliverable that makes debugging agents gradeable at all.
• Property and invariant tests from a stated contract. "Serialising then deserialising returns an equal object", "the result is always sorted", "output length never exceeds input length" — properties are specifications the agent can be told rather than made to infer, and a property test explores far more of the input space than the examples a human would have written. Ask for properties first and cases second; the ordering matters more than it sounds.
• Boundary enumeration around an existing assertion. Given one human-written test that encodes the intent, an agent is reliable at fanning out the edges — empty, null, unicode, off-by-one, maximum, negative zero. The oracle is inherited from the seed test, which is exactly the case where the inference is safe.

Every generated test is a permanent liability on the maintenance ledger.

Generation cost went to nearly zero. Execution cost, review cost, and the cost of a test that fails for the wrong reason did not. A suite is not an asset that accumulates — it is a subscription, billed on every CI run and every future refactor, and the bill is paid by people who did not order it.

• Over-specification is the dominant defect, not incorrectness. Agents assert on everything available: exact log strings, dictionary ordering, private attributes, the precise wording of an error message. Each one is a tripwire that fires on a harmless change. A suite like this does not catch bugs, it catches edits — and it trains the team to disbelieve red builds.
• Flakiness compounds across a large generated suite. One test with a 0.5% failure rate is a nuisance; four hundred of them make a green build impossible and every subsequent agent run unusable, because a background coding agent cannot tell your flake from its own regression. Quarantine on the second unexplained failure, automatically.
• Duplicate coverage is invisible and expensive. Ask three times and you get three tests of the same path with different names. Nothing in review catches this and nothing in coverage reports it; it shows up as a suite that takes eleven minutes to tell you what four minutes told you last quarter.
• Price the runtime before you merge. A hundred generated tests at 200 ms each add twenty seconds to every CI run for every engineer forever. That is a real number and it is worth putting next to the mutation score in the same pull-request comment.
• Delete aggressively, and make deletion cheap. The right disposition for most generated tests is rejection, and a review culture that treats deleting a test as a loss will accumulate the whole ledger. Tests that have never failed and cannot fail are not neutral.

What the agent needs in the loop, and what it must not have.

Test generation is an agentic task with an unusually clean verification signal, which makes the loop design straightforward once you get the boundaries right.

• It must run the tests it writes. A generation step with no execution produces tests that do not compile, import the wrong module, or call a function with the wrong arity — at a rate that makes the output useless. Give it a sandboxed runner and require green-before-submit as table stakes, per sandboxing and execution.
• Feed it the test conventions, not just the source. Fixtures, factories, the project's mocking policy, the helper that builds a valid user. Without them the agent reinvents scaffolding that already exists, and the reviewer's time goes on rejecting a parallel fixture ecosystem rather than on judging the assertions. Two exemplary existing test files in context are worth more than any instruction.
• Withhold the implementation when you want a specification. This is the single most useful lever on the whole page and almost nobody pulls it. Give the agent the signature, the docstring and the issue, and hide the body: it can no longer copy behaviour, so it has to write what the function should do — and where its guess and your code disagree, one of the two is a bug worth looking at. The disagreements are the output.
• Never let it edit the test and the code in the same run. An agent asked to make the suite pass will make the suite pass, and the cheapest edit is usually to the assertion. Separate the runs, separate the diffs, and forbid changes to existing tests in a generation task — this is the same containment argument as in patch generation and test-driven loops.
• Cap the batch at review size. Forty tests in one pull request get skimmed, and skimmed generated tests are worse than no tests because they carry the authority of having been reviewed. Ten, ranked by mutation kills, is a pull request someone will actually read.

Four numbers, and none of them is coverage.

The metric a test-generation tool reports by default is tests written. That number can rise while the suite gets worse, so put these on the dashboard instead.

• Mutation score on the changed lines. The one measure of whether the tests pin behaviour. Track it per pull request, not per repository, so it is a gate rather than a trend.
• Merge rate of generated tests. Merged over proposed. A low rate is information, not failure — it usually says the task class was wrong, not that the model was. A rate near 100% says nobody is reading.
• Escaped-bug rate, before and after. The only outcome measure that matters: production defects per release. If a thousand new tests did not move it, they were coverage theatre, and the honest response is to delete them rather than to generate more.
• Suite wall-clock and quarantine count. The running cost. Both should be flat or falling; if either is climbing with the test count, the ledger is going the wrong way and the team will route around the suite within a quarter.

Related: patch generation & test-driven loops for the other direction of the same loop, evaluating coding agents for how the suite becomes the eval, code review agents for the precision argument applied to review, and the cost of human review for what a forty-test pull request actually costs.