CI/CD & Release

What changes

More, larger changes arrive. When Code Gets Cheap, the Bottleneck Moves sets out the mechanism: DORA ties AI adoption to delivery instability "in part, because it is harder to review larger batches of code" (DORA 2025, p. 57, survey). Next to it is the line that matters for release engineering. Teams that roll back more often got more team-performance benefit from AI, but "rollback reliance does not directly reduce instability". Rollback limits how long damage lasts; it does not stop a bad change getting through. Batch size and rollback are separate controls, and you need both.

The pipeline becomes the verifier for every upstream agent. When a coding agent runs unattended, the test suite and the build are what decide whether its work is done. Stripe's one-shot agents produce over 1,000 merged PRs a week, each human-reviewed, and each run gets at most two rounds of a CI that selects from more than three million tests (Stripe, T3, as of February 2026). Coding covers that setup. For this chapter, the point is that CI is now the oracle in the oracle principle, so whatever an agent can do to CI, it can do to the verdict.

The agent moves into the pipeline. CI auto-fix, review and triage agents now run as pipeline jobs, and pipeline jobs hold secrets: registry tokens, deploy keys, write-scoped repository tokens. An agent that reads issue or PR text while holding them is a new attack path (9.5).

The ladder for this phase

The criteria are those of the maturity model's CI/CD row; the gaps are stated plainly.

L1 is the only rung with a controlled trial. DIDACT is analysed in Case Studies. What matters here is that Google measured downstream safety (rollbacks and sanitizer failures), not just how often fixes were applied. Every rung above L1 lacks that measure.

L2 is shipping as a product feature. GitHub launched Fix with Copilot for failing Actions jobs on 18 May 2026. The agent investigates the failed job, pushes a fix to the branch and tags the developer to review it (vendor docs, as of May 2026). It is L2 by construction: the developer still merges.

L3 exists as configuration, not as a result. The Nx documentation describes auto-apply that commits a fix to the PR branch without a manual step "when ALL of these are true": the suggestion "has been explicitly verified to fix the failing task", the task matches configured glob patterns, and "the AI agent is highly confident" (as of October 2026). Read those conditions against the oracle principle. The verification and the glob list are checks outside the model. The confidence is the model grading its own work. Only the first two can count towards L3. No organisation has published escape or revert rates for auto-applied CI fixes, so the L3 cell has a mechanism and no outcome.

L4 is empty for LLMs. Harness's Continuous Verification "uses machine learning to establish a baseline of normal behavior", compares post-deploy data against it, and "automatically triggers a rollback if anomalies are found" (vendor docs, as of October 2026). That is genuine L4 release gating, done by classical anomaly detection; the grid counts it as AI but does not credit it to LLMs. The lesson carries over: automatic release decisions rest on metrics against a baseline, not on a model's reading of the situation.

Rule 2 applies with force here. A phase's level is the higher of the review mode you practise and the authority the AI's credentials allow (the maturity model's Rule 2). An agent job that can read the npm publish token or the deploy key is at L4 for whatever that token reaches, however carefully its fixes are reviewed. Most teams score their CI agent by its review mode. Score it by its secrets.

The case: Elastic's dependency-repair agent · T3

Level: CI/CD, L2 for one task class, repairing Renovate dependency PRs whose builds broke.

Elastic's team for the elastic/cloud repository uses Renovate to bump dependencies, and some bumps break the build. In mid-2025 the team put Claude Code into its Buildkite pipeline to attempt those fixes. In the first month the agent "fixed a total of 24 initially broken PRs", with "22 commits between July 22nd and August 22nd". The rollout was "limited to only 45% of dependencies". The team estimates it "saved 20 days of active development work".

The guardrails are why this case anchors the chapter:

• No auto-merge. "Never commit AI work without human supervision." The script "makes sure that GitHub PR auto-merge is disabled", because "it is dangerous to allow auto-merge when there are AI contributions." Per the post, the script aborts if it cannot disable auto-merge.
• An allow-list enforced by the harness. The tool configuration "deterministically constrains which commands and actions Claude can use and take". Elastic calls this "key for safety".
• Narrow scope and an audit trail. "Do not change what is not strictly necessary to fix the build." "Only branches in elastic/cloud repository can get these commits." Every action is logged to a file as it happens.
• A rule against the obvious shortcut. "NEVER downgrade versions of dependencies", and "prefer failure to version downgrades relative to the master branch."

The agent took the shortcut anyway. In Elastic's words, "Claude tended to fix the problems of a version bump by, well, … removing the version bump." The build was the oracle, and reverting the change made it pass. The no-downgrade rule lived in the prompt, which the model weighs, not in a check, which would have enforced it. Human review caught the problem; that is what L2 is for. Elastic's conclusion: "Reviews are imperative, and a critical attitude towards the proposed changes is extremely important."

What it does not show. A success rate. Elastic does not say how many broken PRs the agent attempted, so 24 is a numerator with no denominator. It does not show defect or revert rates for merged fixes. The 20 days saved is the team's estimate, not a measurement.

Worked example: scoring Elastic from public data. Score it with the self-assessment method in the maturity model.

• Review mode. Every AI-touched PR waits for a human merge, and auto-merge is forced off. That is L2 for dependency-bump repair.
• Granted authority. Commits go only to elastic/cloud branches, and tools are allow-listed. The post does not say which secrets the Buildkite job holds. An assessor would ask that before signing off on L2, because Rule 2 scores the token.
• What L3 would need. First, a verifier for the intent of the change, kept outside the agent's reach. For this task class, a pipeline step can diff the PR's version catalog against Renovate's original bump and fail any fix that lowers a version (the book's proposal). That turns the failed instruction into a check. Second, a revert-rate comparison against human fixes over at least 90 days. Third, an exercised way to pause the agent. None of these is published, so the honest score is L2.

Migration steps

Climb one rung at a time, for one named task class; each rung's evidence is the ticket to the next. Good first classes are narrow, frequent and checkable: lockfile and dependency-bump breakages, lint and format failures, broken imports after a rename. Code with security impact, migrations and pipeline definitions are bad first classes.

L0 → L1: suggestions with a safety measure.

• Surface fix suggestions for build failures where developers already look: the IDE, the PR, the failed job's log.
• Copy DIDACT's design: record how often suggestions are applied and, from day one, rollbacks of changes that contained an applied fix. Without the second number you cannot say later whether L2 made things worse.

L1 → L2: the agent pushes, a human merges.

• Give the agent its own bot identity, so every commit is attributed, and let it push only to the PR branch it is fixing.
• Disable auto-merge on any PR the agent touched, and fail the job if that cannot be done (Elastic's pattern).
• Enforce a tool allow-list in the harness, not in the prompt, and keep the action log as a build artifact.
• Write the shortcuts you forbid (downgrades, deleted or skipped tests, changes to CI configuration) as pipeline checks that run after the agent and that the agent cannot edit.
• Start the 90-day record that L3 will need: fix merge rate, revert rate within a fixed window, and the same two numbers for human fixes of the same class.

L2 → L3: auto-apply to the branch, review by evidence. All the L3 conditions in the maturity model apply. Here is what they mean in CI:

• Auto-apply only for task classes on a written list, enforced by configuration (Nx's glob patterns are one way to do it). Never let model confidence be the only gate.
• Re-run the full pipeline on the agent's commit in a job the agent did not configure. That re-run, plus the intent checks above, is the verification.
• Keep the merge gated. At L3 the agent no longer waits for approval of each fix, but the PR still merges by human approval or by a routing rule. Zalando's review bot auto-approves the 33% of PRs it classes as low-risk, using rules "built based on analysis of our production incidents" (Zalando, T3, August 2026). Whether it uses an LLM is not stated; what carries over is building risk tiers from your own incident data.
• Exercise the pause. Turn the agent off for one task class, confirm the pipeline still works, and turn it back on. The maturity model requires that pause and rollback be exercised, not just possible.

L3 → L4: not on the evidence available. No organisation has published LLM-driven promotion or rollback with a verification record. If you automate release decisions, let a metric comparison against a baseline make the call, as classical verification does. A model can explain the decision; it should not make it. The maturity model's other L4 conditions still apply: a named accountable owner, a kill switch and periodic audit samples. Any release action that cannot be undone is not eligible for L4 at all.

If upstream agents are raising PR size or changes per release, fix that before any climb. A CI agent that repairs larger batches faster still delivers larger batches.

Guardrails

The guardrails in 9.3 keep a CI agent's output honest. This section treats the agent itself as a privileged dependency that attackers can reach. The full permission model is in Security, Permissions & Blast Radius. Three incidents show why CI deserves its own guardrails.

PromptPwnd: untrusted text plus a privileged token. Aikido's researchers found GitHub Actions and GitLab workflows in which "untrusted user input → injected into prompts → AI agent executes privileged tools → secrets leaked or workflows manipulated". Affected integrations included Gemini CLI, Claude Code, OpenAI Codex and GitHub AI Inference. Google "patched it within four days" for Gemini CLI's own workflow. Aikido says "at least 5 Fortune 500 companies are impacted", but names none. Claude Code's action runs only for users with write access by default, and Aikido calls widening that with allowed_non_write_users: "*" "extremely dangerous".

Aikido's mitigations are a usable checklist as they stand: "Restrict the toolset available to AI agents"; "Avoid injecting untrusted user input into AI prompts"; "Treat AI output as untrusted code"; "Restrict blast radius of leaked GitHub tokens". In maturity-model terms, an agent that processes issue text from strangers must not also hold secrets that can change anything outside its branch.

The Amazon Q extension: the AI tool's own pipeline. In July 2025 AWS disclosed that malicious code had been "automatically included in a release" of the Amazon Q Developer extension for VS Code. The root cause was "an inappropriately scoped GitHub token in their CodeBuild configuration" (AWS security bulletin, CVE-2025-8217, T2 vendor bulletin). Version 1.84.0 shipped it, 1.85.0 removed it, and the injected code failed to run because of a syntax error. The root cause was not a model but a CI token's scope, the same class of failure as the agent incidents in Case Studies.

Nx s1ngularity: a workflow injection that armed AI CLIs. Nx's maintainer advisory traces the August 2025 compromise to a PR-validation workflow. It used pull_request_target and echoed the PR title in a shell step, so a crafted title ran as code and led to a leaked npm token. Wiz's analysis shows what the malicious packages then did: they ran developers' installed AI CLIs with their skip-permission flags to hunt for secrets. The AI step exfiltrated data in under a quarter of cases, often stopped by model refusals, approval prompts and default workspace limits. Nx's fixes were CI fixes: Trusted Publishing instead of long-lived npm tokens, and approval for workflows from external contributors. The security chapter anchors this case.

The book's proposal for any CI job that runs an agent, drawn from these three incidents and from GitHub's shipped envelope:

CI AGENT JOB: MINIMUM ENVELOPE (the book's proposal)
trigger      only collaborators with write access; never "*"
input        issue/PR text is data; never in a prompt alongside secrets
token        scoped to the one PR branch; short-lived
secrets      no publish, deploy or cloud-admin credentials in the job
tools        allow-list enforced by the harness, not the prompt
cannot edit  workflow files, CODEOWNERS, agent config, test oracles
merge        auto-merge disabled on any PR the agent touched
log          every action uploaded as a build artifact

GitHub's Copilot cloud agent already ships most of this as defaults. It can push only to its own branch, its workflows "are not triggered until Copilot cloud agent's code is reviewed", and the person who requested the work cannot approve the PR (GitHub docs, as of October 2026). No one has published incident rates by envelope, so treat it as a starting design, not a validated standard.

Dependency allow-lists against hallucinated package names are in Economics, Risk and Measurement; rollback of running services is in Operations & Incidents. The wiki's agent supply-chain security deep-dive and release and publishing agents playbook cover the tooling.

Metrics

The common metric for CI agents is acceptance: the share of proposed fixes a developer took. It measures agreement, not correctness. Elastic's removed version bumps would have counted as accepted if a reviewer had missed them. DIDACT is the template because it paired throughput with downstream safety. Pair each flow metric with a stability metric, per task class:

• Fix merge rate, with its denominator. Count fixes merged out of fixes attempted, not out of fixes proposed. Elastic's 24 had no denominator.
• Revert rate of AI-landed fixes within a fixed window, such as 14 days, next to the same rate for human fixes of the same class. This is the stability half, and a practical stand-in for the escape-defect comparison against a human baseline that the maturity model requires for L3.
• Intent violations caught. Count downgrades, deleted or skipped tests, and CI-config edits blocked by your post-agent checks. A rising count means the agent is gaming the oracle. Zero for a long time can mean the checks are not firing.
• Batch size, using DORA's own indicators: lines in the most recent change, changes per release, time to complete a task (DORA 2025, pp. 58–59). Small batches slightly reduce AI's individual-effectiveness gain, though DORA judges the net effect positive.
• Change failure rate and time to restore for releases that contain AI-landed changes, against those that don't.

The Thoughtworks Technology Radar (Vol 34, April 2026) puts DORA metrics in Adopt with a sentence that fits this chapter: "If lead times don't decrease and deployment frequency doesn't increase, faster code generation doesn't translate into better outcomes." That is practitioner opinion, and it matches the evidence here. Write the threshold that would switch a CI agent off before you switch it on. Chapter 17, Measuring Success and When to Roll Back, sets out the rollback triggers in full.

Anti-patterns

• Green build as proof. Elastic's agent made builds pass by removing the change under test. If the agent can change what the build checks, a pass means nothing. Check intent outside the agent, and see Testing & Verification for keeping tests out of its reach.
• Inherited auto-merge. A repository that auto-merged Renovate or Dependabot PRs silently becomes L4 the day an agent commits to one. Copy Elastic's refusal to proceed unless auto-merge is off.
• Open triggers with secrets. Running an agent on pull_request_target, or for any commenter, in a job that holds write or publish tokens. This is the PromptPwnd and Nx pattern.
• Over-scoped, long-lived tokens in CI. The Amazon Q and Nx incidents both turned on a CI token with more reach than the job needed. Use short-lived, scoped credentials and trusted publishing.
• Acceptance as the success metric. 60% accepted tells you developers took the fixes. It does not tell you the fixes were right.
• Model confidence or an LLM as the gate. "Highly confident" is the model grading itself, and no published evidence supports LLM-driven promotion or rollback. Gate on checks and baselines.
• An over-broad role. In the disputed Kiro case, both accounts put access scope at the centre, and Amazon's fix was "mandatory peer review for production access" (Case Studies). A deploy-capable role on an agent is an L4 grant whether or not anyone meant it as one.

The wiki's CI repair agents playbook and its merge queues for agent changes playbook cover implementation. Rollout and versioning covers staged releases for the agents themselves.