8 min read

U34
Playbook · Coding & Computer-Use Agents

Flaky-Test Triage Agents

Proving a test is flaky is cheap; proving one is fixed is not. Build the agent around that asymmetry or it will quietly delete your suite.

A flaky-test agent that can quarantine but never has to justify a release will shrink your suite to the tests that always pass, and every build will be green on the way down. The asymmetry is the whole design: one rerun that passes on an unchanged commit proves flakiness outright, while proving a test is no longer flaky takes 59 clean reruns at a 5% flake rate — so quarantine entry can be automatic and quarantine exit can only ever be a deadline with a name on it.

Step 1Do the arithmetic before you design the loop, because it decides which questions the agent may ask.

A test that fails independently 5% of the time passes a single rerun 95% of the time. Rerun it n times and see nothing, and the chance you would have seen nothing anyway is 0.95^n. To push that under 5% — the weakest confidence anyone would accept — takes 59 consecutive passes. At a 1% flake rate it takes 299. Those numbers are the reason "just rerun it until we know" is not an available strategy: the agent would spend more compute confirming one test's health than running the entire suite.

Now run it the other way. A test that failed and then passed on the identical commit, with no code change between the two runs, is flaky by definition — the suite has just contradicted itself, and one observation settles it. Flakiness is cheaply provable and health is not, and every routing decision in the agent follows from that:

  • "Is this failure a flake?" — answerable, cheaply, from one rerun plus history. Let the agent decide this and act on it.
  • "Is this test healthy again?" — not answerable at any price you will pay. Never let the agent decide this. Exit from quarantine is a date and an owner, not an evidence threshold.
  • "Is this a real regression?" — answerable only in the negative. The agent can show a failure is a flake; it cannot show a failure is not, so the default on an unresolved failure must stay "blocks".

The common build inverts this. Teams give the agent a confidence score and a threshold, and the threshold silently answers the unanswerable question — "80% confident it is flaky" becomes a de-quarantine, and the 20% is a regression now shipping on a green board. See eval variance and statistical power for the same error in eval harnesses.

Step 2Feed it history keyed to test identity, not to the test's name.

The agent's entire diagnostic power is in the record of previous runs, which means the record has to survive the things that happen to tests. A rename, a move between files, or a parameterised case whose id contains a timestamp all break name-keyed history, and the agent then sees a brand-new test with a single failure and no basis for any judgement at all. Key on something stable — a hash of the test's fully qualified path plus its parameter set, carried in the report — and store, per run: the commit, the outcome, the duration, the worker or shard, and the failure's first stack frame.

# The four queries that do all the work. Everything else is decoration.
same_commit_disagreement  # failed and passed at the same SHA -> flake, proven
first_seen_failing_sha    # bisect target if it has never passed since
shard_skew                # fails only on worker 3 -> shared port/fixture, not the test
duration_p99_vs_timeout   # 'flake' that is a timeout you set 200ms above p99

That last query catches the single most misclassified family in practice. A test whose p99 duration sits just under its timeout is not flaky, it is correctly reporting that the thing it calls got slower, and an agent that quarantines it has suppressed a performance regression. Make the agent check the duration distribution before it is allowed to use the word flake at all — see performance optimization agents for where that signal should have gone instead.

Step 3Make quarantine a governed action with an expiry, not a label.

Quarantine is the agent's only real power and it is a write to your risk posture, so it gets the treatment any other privileged action gets: a record, a named owner, a blast radius and a deadline that fires whether or not anyone did the work. The expiry is the load-bearing field. Without it the population only grows, because nothing in the system has an interest in taking a test back out.

# quarantine/checkout_total_rounding.yml — written by the agent, owned by a human
test: tests/billing/test_checkout.py::test_total_rounding[eu]
identity: 9f2c4b           # stable hash, survives renames
proven_by: same-commit-disagreement at 4e81aa0
first_seen: 2026-11-14
owner: billing-platform    # a team that can be paged, never 'unassigned'
expires: 2026-11-28        # 14 days. On expiry the test un-quarantines and blocks.
scope: pr-gate             # pr-gate | nightly | never-release-gate
still_runs: true           # reported, just not blocking. NEVER skip the test.
suspected: shared-fixture  # agent's hypothesis, not a verdict

Two fields deserve argument. still_runs: true is non-negotiable: a quarantined test must keep executing and keep reporting, or you have lost the only data that could ever retire the quarantine, and you have converted a known-unreliable signal into no signal. And scope exists because the release gate is a different question from the pull-request gate — an agent may reasonably stop a flake from blocking one developer's merge and must never be the thing that decides a release ships. Wire the gate itself as described in merge queues for agent changes.

Step 4Give it the four repairs that are real, and forbid the one that works.

Most flakiness has a small number of mechanical causes, and an agent with repository context is genuinely good at them because each leaves a recognisable signature in the diff or the failure. Scope the agent to these and require a reproduction before and after:

  • Order dependence — passes alone, fails after a sibling. Signature: state left in a module-level or class-level fixture. The repair is isolation, and it is verifiable by running the pair in both orders.
  • Shared resource collision — fails only under parallelism or only on one shard. Signature: a hard-coded port, path, table name or temp directory. The repair is to derive it from the worker id.
  • Clock and timezone — fails at midnight, on the 1st, or in one CI region. Signature: a real now() in the assertion path. The repair is to inject the clock.
  • Unconditional waits — a sleep standing in for a condition. Signature: a timing-shaped failure under load. The repair is to poll the condition with a timeout, which is a different thing from sleeping longer.

Then the prohibition. The agent must never add a retry wrapper, lengthen a sleep, loosen an assertion, or mark a test skipped — and the reason is not purity, it is that retry-on-failure is the most effective of all these repairs and the arithmetic is what makes it a trap. At a 5% flake rate, retrying a failure up to three times drops the reported failure rate from 1 in 20 to roughly 1 in 8,000, and costs about 5% more runtime, because the extra runs only happen on the 5% that failed. It is cheap, it works, and it is indistinguishable at the dashboard from having fixed the test. It also hides a genuine 5%-of-the-time regression just as completely. A repair the agent is allowed to make must change the test's behaviour, not its reporting.

If your CI already retries failures globally, the agent has no signal to work from — same_commit_disagreement is being consumed and discarded inside the runner before anything records it. Turn the retry into a recorded rerun that reports both outcomes before you build any of this. That one change is often worth more than the agent.

Step 5Measure the population, not the agent's close rate.

A close rate rewards exactly the behaviour you are trying to prevent, because quarantining is a close. Four numbers tell you whether the thing is reducing risk or laundering it, and all four are properties of the suite rather than of the agent:

  • Quarantine population over time — must be flat or falling. Rising monotonically means the expiry is not firing or is being renewed by default, which is the same failure.
  • Median quarantine age — the honest measure of whether anyone is doing the repairs. If it exceeds your expiry window, your expiries are being rubber-stamped.
  • Escape rate — defects that reached the default branch through a test that was quarantined at the time. This is the number the agent exists to keep at zero, and the only one that can justify shutting it off.
  • Reruns per proven flake — your actual compute bill for the diagnosis. If it drifts above a handful, the agent is rerunning instead of reading history, and STEP 1's arithmetic is about to arrive as an invoice.

Report all four on the same surface as the suite's pass rate, because the pass rate alone will improve under every one of the failure modes above. Quality regression detection covers the general shape of this trap.

Step 6Three cases where the agent is the wrong instrument.

Your flakiness is concentrated. If twenty tests account for most of the noise, a week of one engineer's time removes the problem permanently and an agent institutionalises it. Count first: a Pareto check costs an afternoon and frequently cancels the project.

The flakiness is in the environment, not the tests. A shared staging database, a rate-limited third-party sandbox, or a CI runner that oversubscribes CPU produces flakiness that is uniform across unrelated tests. The agent will quarantine dozens of innocent tests and never touch the cause. The shard_skew and cross-test correlation checks exist to detect precisely this and hand it to infrastructure instead — see CI repair agents.

You cannot un-quarantine. If no team owns the suite, nothing will ever come back out, and the expiry will be renewed forever by whoever is on call. An agent that can only add to a list is a suite-deletion machine with a 14-day delay.

Do this today, before writing any agent: make your CI record both outcomes of every rerun against the same commit, and emit a stable test identity in the report. Then run one query — same_commit_disagreement over the last 30 days — and look at the distribution. You will learn whether you have a flakiness problem or twenty bad tests, and you will have built the only dataset the agent would have needed. Related: patch generation and tests, reproducibility and determinism, and retry amplification.