CI Repair Agents

Four classes, and each one needs different evidence.

A red pipeline is not a bug report. It is an observation that admits four incompatible explanations, and the entire value of the agent is in choosing correctly between them before it writes a line:

• Caused by this diff. The change under test broke something. This is the only class where the agent should be editing the change itself, and it is the class everyone assumes by default.
• Pre-existing on the base. The same check is red on the branch you merged from, so the diff is innocent and a fix pushed here is a fix in the wrong pull request. The correct output is a note naming the base commit and, when a fix already exists elsewhere, porting it rather than waiting.
• Environment or infrastructure. Checkout failed, the registry timed out, the runner was reclaimed, a container image moved. No test body ran, so there is no assertion to reason about, and generating a code change from this input is pure confabulation.
• Non-deterministic. The test passes and fails on identical input. This is a real defect in the test, and it is also the class an agent under pressure to produce something will reach for whenever it cannot explain a failure — which is why it needs the strictest evidence bar of the four.

Prove causation before you push, because the check is cheap and the wrong push is not.

The evidence that separates class one from class two is mechanical and nobody runs it: execute the failing check against the merge base as well as the head. Two runs, and the result is decisive — red on both means the diff is innocent; green on base and red on head means it is not. On a repository with a merge queue you have the base result already, sitting in a build somebody else's pull request triggered an hour ago, and the only work is looking it up.

• Reproduce before you fix. The agent should be able to make the failure happen on demand — the same discipline patch generation and tests insists on. If it cannot reproduce, it has not localized, and a patch built on a log excerpt is a guess wearing a diff's clothing.
• Prove the fix, both directions. The failing check must fail before the patch and pass after it, on the same commit and the same runner. "The suite is green now" is compatible with the agent having changed something unrelated while the real cause resolved itself.
• Price the alternatives honestly. Two extra CI runs cost minutes of compute. A wrong push costs a full pipeline, a stale review, an approval reset on a pull request a human had already signed off, and a permanent increment to how much the team discounts the next thing the bot says. The asymmetry is enormous and it is the whole argument for spending the runs.

"Flake" is not a root cause, and the re-run budget is one.

Re-running a failed job is the most available action the agent has and the most easily abused. A single passing re-run establishes exactly one thing: the failure is not deterministic on this commit. It does not tell you the test is at fault rather than the code, and it certainly does not license closing the incident — a genuine race condition in the product presents identically to a flaky assertion in the test, and the second one is the comfortable conclusion.

• One re-run, and only for the cases it can decide. Spend it when the job died before any test body executed, when the same commit passed earlier, or to confirm a suspected base-branch failure. A second failure is real; treat a second re-run as a policy violation rather than a retry.
• Never skip, disable, quarantine or loosen a test to reach green. This is the single highest-value guardrail on the page, because it is the shortest path to green and an agent optimising for green will find it. A deleted assertion is one of the three honest liars in the loop, and the deletion is invisible in a diff that also contains a real fix.
• Never push an empty commit or reopen a pull request to kick CI. Both are ways of re-rolling the dice while producing an artefact that claims work was done.
• Repair the flake when it is in scope. Non-determinism has causes — shared fixtures, wall-clock assertions, unordered collections, a port bound by a leftover process — and a fix with a named cause is a legitimate deliverable. A quarantine annotation is not.

What the agent may edit follows from the class it chose.

Authority should be a function of the classification and the evidence behind it, not a global setting. The failure mode this prevents is the agent widening a pull request to fix something real but unrelated, which is how a two-line change becomes unreviewable:

• Class one, in code the diff touches — fix and push. This is the case the agent exists for and the only one where its authority is uncontroversial.
• Class one, in code the diff broke but does not touch — fix and push, keeping the change minimal. A downstream caller broken by a signature change is still your diff's fault.
• Class two, with a fix that exists — port the fix, push, and say what you did. It no-ops once the base carries it, and waiting for someone else's pull request to merge is still waiting.
• Class two with no fix, or class three — do not push. Write down which check failed, on which base commit, and what evidence establishes it is not this change. A comment naming the blocker is a result; silence is not.
• Generated files, lockfiles, snapshots — regenerate with the repository's own tooling and never by hand, and treat a hand-edited lockfile as a failed run regardless of whether CI goes green.

The boundary that holds all of this together is that the agent may change the code under test or the test's own defects, and may never change what the test asserts about behaviour. The moment those two are confused you have an agent that certifies bugs, which is the failure test-generation agents run into from the other direction.

You are racing a moving branch, so the fix has to be idempotent.

Unlike a debugging agent working against a fixed production incident, a CI-repair agent operates on a branch other people are pushing to and a base that moves under it. Three things follow, and all three are operational rather than intellectual:

• Re-verify the head before you push. The failure you diagnosed may already be fixed by a commit that landed while you were working, and pushing on top of it produces a confusing no-op or a conflict. Read the current head, not the one that triggered you.
• Never rewrite history on a branch you do not own. No rebase, no amend, no force-push — a colleague's checkout is a real thing you can break, and a merge commit costs nothing by comparison. On a branch the agent created, follow whatever the repository's contributing guide says.
• Deduplicate on the failure, not on the event. CI emits the same failure repeatedly and webhooks arrive more than once, so key the agent's work on (check name, head commit) and make a second delivery a no-op. Without this you get two agents pushing two fixes for one failure, which is the idempotency problem in its most embarrassing form.
• Cap the loop. Fix, push, fail, fix, push is a loop with no natural end. Give it a hard attempt ceiling per pull request and make exceeding it an escalation with the diagnosis attached, per planning and termination — three failed attempts is information, and continuing past it burns both CI capacity and budget.

Measure the wrong pushes, not the green builds.

Builds turned green is the vanity metric, and it is worse than useless because the fastest way to move it is the behaviour you most need to prevent. The numbers that describe whether this agent is helping:

• Wrong-push rate. Pushes that did not fix the failure, or that fixed a failure the diff never caused. This is the number the team feels, and it is the one that decides whether anyone reads the agent's comments in three months.
• Classification accuracy, sampled by humans. Take twenty failures a week and label them independently. Confusion between class two and class four — a base-branch breakage called a flake — is the specific error to watch for, because it is the one that lets a real regression sit on main.
• Time from red to a correct diagnosis, separately from time to green. Diagnosis is the product; the patch is sometimes not needed at all.
• Re-runs spent per failure. If this creeps above one, the agent has discovered that re-running is cheaper than thinking.
• Assertions deleted or tests skipped in agent-authored commits. Should be zero, should be alerted on, and should be checked mechanically rather than at review time.