7 min read

U3
Playbook · Coding & Computer-Use Agents

Patch Generation & Test-Driven Loops

The patch is a hypothesis and the suite is the experiment — but the suite is also an artefact under test, and on the field's own benchmarks roughly a third of passing patches are not actually verified by the tests that passed them.

A green test run means "nothing that ran failed", and the gap between that and "the patch is correct" is now measured. On SWE-bench Verified, a 2026 study found that 7.8% of patches counted as correct while failing the project's own developer-written tests, and differential testing put 29.6% of plausible patches at behaviour differing from the ground-truth fix. An earlier manual review of successful patches on full SWE-bench attributed 31.08% of passes to weak tests and 32.67% to the solution being sitting in the issue text. Your patch loop is only as discriminating as the oracle it optimises against — so the step most loops are missing is the one where the agent strengthens the test before it is allowed to call the patch done.

Step 1Edit by structured diff, not by rewriting files.

Full-file rewrites are token-expensive and silently destructive — the model drops a function it did not "see" as relevant. The robust unit is a localized hunk: a search/replace block or a unified diff anchored on context lines. This forces the model to commit to exactly what changes, makes the edit reviewable, and turns a botched edit into a clean, recoverable apply failure instead of a corrupted file the agent then has to debug as if it were the bug. It also makes the diff the unit of review for a human, which is the only form in which a reviewer can realistically audit agent output at volume — see review queues for agent output.

Step 2Hunk-apply failure is information, not just an error.

When a hunk does not apply, the cause is almost always that the model's idea of the file is stale — wrong line numbers, drifted context, an earlier edit it forgot. The agent must treat the rejected hunk as a signal to re-read the current file state, not retry the same diff harder. Production agents respond to apply failures by re-opening a tight window around the target, regenerating the hunk against the actual bytes, and only then re-applying. Retrying the identical diff is the degenerate case of retry amplification: it cannot succeed, and it costs a full model call each time.

# apply -> test -> read -> revise, with honest failure handling
try:
    repo.apply(hunk)
except HunkReject as e:
    window = repo.open(e.file, e.line, ctx=40)  # re-ground on real bytes
    hunk   = agent.regen(window)               # not: retry same diff
res = sandbox.run_tests(scope="changed")        # fast loop: targeted first
if res.passed: res = sandbox.run_tests(scope="full")  # then guard regressions

Run the targeted test first for a fast revise loop, then the full suite as a regression gate before submit. Skipping the full pass is how a green targeted test ships a broken neighbor.

Step 3Reproduce before you fix — and distrust a repro you were handed.

The strongest pattern across SWE-agent and OpenHands runs is TDD inverted onto the agent: write or run a test that reproduces the bug first, watch it fail for the right reason, then edit until it passes and the rest stay green. A failing repro converts a vague issue into a concrete oracle and a stack trace that localizes for free. An agent that edits before it has a red test is optimizing against a target it cannot see.

The 2026 evidence adds a second instruction that is less intuitive: be suspicious when the repro arrives too easily. In Aleithan et al.'s manual review of successful SWE-agent patches, 32.67% of the passes came from the solution being present in the issue report or its comments — the agent was transcribing, not diagnosing. That is a benchmark artefact, but the same shape occurs constantly in real work: a ticket that contains a suggested diff, a Stack Overflow answer pasted into the description, a colleague's guess in a thread. The discipline is to let the issue text supply the symptom and derive the repro from the running system, then check whether the mechanism you found matches the one the ticket asserted. When they disagree, the ticket is usually wrong and the patch that follows it is a plausible fix to a bug nobody had.

Step 4Regression guarding: the suite is two oracles, and you must baseline both.

The targeted test answers "did I fix it". The pre-existing suite answers "did I break something else" — a different, equally load-bearing question. Many submitted patches resolve the issue and silently fail a neighbor; on the SWE-bench family the difference between a model-shaped patch and the gold patch is frequently a regression, not a missed fix. The discipline: diff the pass/fail set against the pre-patch baseline, and treat any newly-red test as a hard block, even if the targeted test is green.

Beware the suite that was already partly red. The agent must baseline failures before editing; otherwise it will chase pre-existing flakes, "fix" unrelated tests, or declare victory because a test that never passed still does not. Store the baseline as data the agent reads, not as something it is told in a prompt — a prompted baseline drifts out of date within one session.

Step 5Measure the oracle's discriminating power, because "coverage" does not.

This is the step almost no loop has, and it is where the 2026 numbers point. Wang, Pradel and Liu examined plausible patches from three tools on SWE-bench Verified — the hand-curated subset specifically built to be trustworthy — and reported that the benchmark's validation mechanism lets 7.8% of all patches count as correct while failing the developers' own test suite, and that automated differential testing found 29.6% of plausible patches behaving differently from the ground-truth patch. Of the suspicious patches they sampled, 28.6% were definitively incorrect on manual inspection and, tellingly, for 66.2% they could not determine correctness at all from the available information. A follow-up line of work on adversarial test generation reports strengthening 50.2% of SWE-bench Verified instances — which is another way of saying half the instances had tests weak enough to strengthen.

Line coverage will not show you any of this, because coverage measures execution, not discrimination: a test that calls the function and asserts nothing covers it completely. The measurement you want is whether the tests that must pass can tell a correct patch from a wrong one, and there are two cheap ways to get it. Mutation on the touched files: inject small semantic changes into the lines the patch modified and check how many the suite kills; a mutation that survives is a wrong patch the loop would have accepted. A known-bad patch set: keep a handful of deliberately wrong fixes per critical module and assert the suite rejects them, exactly as you would meta-evaluate a judge in judge calibration. Both give you a number per module, and that number — not coverage, not pass rate — is how much a green run is worth there.

Then make the loop act on it. On a module whose discriminating power is low, the correct next step after a green targeted test is not to submit: it is to have the agent add the assertion the suite was missing, show that the new test fails on the pre-patch code and passes on the post-patch code, and submit both. That ordering also defends the loop against itself, because a test written after a patch tends to encode the patch; demanding that it go red on the old code is what keeps it an oracle rather than a description. Test-generation agents are the right tool and the wrong default — generated tests raise coverage readily and discrimination only when you force the red-then-green proof.

Step 6The loop's three honest liars, and the ceiling that is now a published number.

Three failure modes corrupt the signal. Flaky tests make a correct patch look broken and a broken one look fixed — quarantine and re-run before trusting a flip. Overfitting: the agent special-cases the exact fixture inputs instead of fixing the mechanism, passing the visible test and failing every held-out one. Reward hacking the oracle: weakening an assertion, deleting the failing test, or wrapping the call in a try/except that swallows the error — technically green, substantively a lie, and the reason test files should be a separate reviewable surface in the diff rather than ordinary lines the agent may edit freely. All three pass the loop and fail reality; the mechanism behind the third is reward design and hacking.

And the ceiling is no longer a matter of opinion. Epoch AI's benchmark review programme rated SWE-bench Verified "Flawed" in a review dated 3 September 2026, under a published threshold — a benchmark is Flawed when scoring defects could affect at least 20% of the full score. The benchmark the whole field reports against is, by that standard, not a reliable oracle; the argument and the rest of the audit is in the tasks were public, so were the defects. If a curated academic suite built for exactly this purpose has defects affecting that share of its score, your repository's suite — written for humans, by humans, under deadline — is not better.

The change to make this quarter is one measurement and one loop step. Run mutation on the files each agent patch touches and record the kill rate per module; then make a low kill rate block auto-merge and require the agent to ship a strengthening test that goes red on the pre-patch code. A passing suite proves the patch did not break what was tested — never that it is correct. The loop's ceiling is the suite's discriminating power, and unlike coverage, that is a thing you can raise on purpose.