Coding

What changes

At L1 the unit of AI work is a keystroke: a completion accepted, or a snippet pasted from chat into a commit a human still writes. At L2 and above it is a whole pull request. That moves the coding phase's job. When an agent writes the diff, typing is no longer the scarce resource. The scarce resource is an engineer's ability to tell, quickly and reliably, whether the diff is right. The bottleneck chapter traced where that effort goes across the delivery system. This chapter covers the part that sits inside the repository. How much faster assisted coding makes an individual is a range, from 19% slower to 55.8% faster depending on task type, familiarity, codebase maturity and tool generation, and the economics chapter reads it. The question here is narrower: what has to be true of a repository, a task and a toolchain before an agent's pull request is cheap to check?

The best-documented answer is that the check has to exist before the agent starts. Anthropic's Claude Code best-practice guide (vendor guidance, read October 2026) puts it in one line: "Give Claude a check it can run: tests, a build, a screenshot to compare. It's the difference between a session you watch and one you walk away from." The reason follows: "Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available, and you become the verification loop." Swap the product name and the point holds for any coding agent. An agent without a runnable check needs a human beside it, which is L2 at best. An agent with one can be left alone, the first precondition for L3.

Two further shifts follow. The repository becomes an input to the agent: its build, test commands, conventions and forbidden zones either reach the agent or they don't. And coding can now outrun the phases that check it. The maturity model's Rule 6 heuristic (the book's proposal, on correlational data) says to run coding at most one level above the lower of review and testing. Most teams are early on this path. In the Stack Overflow 2025 developer survey (survey, self-selected), 14.1% of respondents used AI agents daily at work, and 37.9% did not plan to use them at all.

The ladder for coding

The criteria are those of the maturity model's coding row. Here each cell also gets the reason it sits where it does. A level is scored per named task type. It is the higher of the review mode you practise and the authority the agent's credentials allow.

L0 and L1. L0 is a choice, not a lag. A codemod or dependency bot opening thousands of pull requests is still L0 for AI. L1 is where most measurement lives, and its lesson for this chapter is a negative one. An assistant that makes one engineer faster at typing code says nothing about whether an agent's whole pull request will pass review.

L2 is the realistic level for most teams, and its evidence shows a human still doing real work. The dotnet/runtime log (T2) is the fullest public record, and the case studies chapter reads it in detail. Two of its numbers define L2 better than any definition. When a human pushed commits to the agent's branch, 86.2% of those pull requests succeeded. Without human commits, 55.1% did; the post cautions that this partly reflects exploratory pull requests deliberately left alone. And merged agent pull requests drew 16.5 review comments on average, against 12.4 for human ones. The agent produces the artefact; the human steers and corrects it. An independent study of Claude Code pull requests in open source (Watanabe et al., T1) agrees. Of 567 pull requests across 157 projects, 83.8% were merged, but only 54.9% of those merged without further modification. The agents were used mainly for refactoring, documentation and tests, the bounded task types this chapter returns to.

L3 exists in production, but only as volume. What changes from L2 is that nobody watches the run. The agent works unattended in an isolated environment, and a human sees only the finished pull request. Stripe is the clearest public example and is this chapter's case. GitHub's Copilot cloud agent (vendor docs, as of October 2026) ships the matching permission envelope. Only users with write access can trigger it. It can push only to its own branch. Its workflows "are not triggered until Copilot cloud agent's code is reviewed", and the person who asked for the change cannot approve it. Faros reports agents opening 13–14% of pull requests "at the leading edge" (Faros, vendor telemetry, T3, as of September 2026); take the direction from that, not a share. The failure case is Answer.AI's month with Devin, where an autonomous agent given loosely scoped real tasks succeeded on 3 of 20. None of these sources publishes an escaped-defect or incident rate for the unattended work against a human baseline. The model requires that comparison at L3, so every public L3 claim in coding is a claim about design, not outcome.

L4 has no LLM evidence. Nobody has published auto-merge of LLM-written code for a task class with a verification record. Spotify's Fleet Management auto-merges most of its 2.5 million-plus pull requests (as of June 2026), but the system began as deterministic scripts and Spotify does not say how many of the auto-merged ones its LLM agent wrote. Where an agent really did act without per-item approval, it held a credential nobody meant to give it, as with Replit's and PocketOS's lost production databases. Those are de facto L4 grants, scored by authority, not intent.

The case: Stripe's Minions

Stripe's engineering post "Minions: Stripe's one-shot, end-to-end coding agents" (Alistair Gray, 9 February 2026) describes an unattended coding-agent deployment at scale, with its design in public. The headline is "Over a thousand pull requests merged each week at Stripe are completely minion-produced." The qualifier sits in the same passage: "while they're human-reviewed, they contain no human-written code."

Level: Coding, L3 by design for the routine task types it handles. Runs are unattended and sandboxed, and a human reviews every resulting pull request. Stripe is at L3 because nobody watches the run, not because review was removed. It wasn't.

Read the post for its preparation, not its volume. Almost every design choice is one of this chapter's three ingredients.

• The environment is built for the agent. Minions run in devboxes that "are pre-warmed so one can be spun up in 10 seconds" and are "isolated from production resources and the internet". Because of that isolation, minions run without per-action permission checks. The sandbox does the job that approval clicks would do badly.
• The agent reads the instructions humans' tools read. "Minions read the same coding agent rule files that human-operated tools such as Cursor and Claude Code do." No separate prompt is kept for the unattended agent. Tools arrive through a central MCP server, Toolshed, with over 400 tools (as of February 2026).
• Checks are fast, deterministic and bounded. Local lints run on each push in "less than five seconds". CI draws on "over three million" tests. A run gets "at most two rounds of CI". That cap is a design decision. An agent that cannot get green in two rounds hands back instead of spending days on the wrong problem.
• Work arrives as named, routine tasks. Engineers "most frequently start one from Slack", and runs also start from a CLI, a web interface and flaky-test tickets. A flaky-test ticket is a task type with its definition of done built in.

What it does not show. Any outcome. With no success rate, you cannot tell how many runs failed for each one merged. There is no revert or incident rate, no review time per pull request and no share of Stripe's total pull requests. "Over a thousand a week" shows the pattern runs at scale. It does not show that it beats the alternative, or what it costs in reviewer hours.

Read it against two Part I cases. Answer.AI's Devin trial used the same architecture class, an unattended agent working to completion, and failed on 14 of 20 tasks; its agent "would spend days pursuing impossible solutions rather than recognizing fundamental blockers." A capped run and a routine task intake are the kind of controls that trial did not have. dotnet/runtime shows the lesson from the reviewed side. Success nearly doubled when the team fixed the environment and wrote instructions, "not through better AI models, but through better preparation". The post closes on this chapter's argument: "If there's one lesson from this experience, it's this: preparation matters more than the model."

Migration steps

Climb one level at a time, per task type. The order matters: the environment comes before the instruction file, and both come before any judgement about the model.

From L0 to L1: decide the stance

Write down whether AI-assisted code is allowed, in which repositories, and with what disclosure. A clear and communicated AI stance is one of the enablers that gate a level. A team that drifts into L1 without one will drift into L2 the same way. Chapter 15, Scaling Across Teams, covers the governance.

From L1 to L2: prepare the repository

1. Make the environment work for the agent first. Run the agent where it will really run and watch where it fails before it touches code. In dotnet/runtime the agent at first could not download the packages its own build needed. The fix was configuration: "We configured the firewall rules to allow access to the package feeds our build needs", plus setup steps so builds did not time out. Pull requests created before those changes succeeded 38.1% of the time, and those created after, 69%. Monthly success went from 41.7% in the first month to about 71% in the most recent quarter. The post cannot separate the instruction file's share of that gain from the environment fixes shipped with it, so fix both and credit neither alone.
2. Write a short instruction file, by hand. The evidence on instruction files (AGENTS.md, CLAUDE.md, copilot-instructions.md) is mixed, and the mix is the lesson.
   • The ETH Zurich study "Evaluating AGENTS.md" (Gloaguen et al., T1) tested agents with LLM-generated context files on SWE-bench tasks, and with developer-committed ones on real issues. It found that "providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average". Agents followed the instructions, but repository overviews did not help. The files "are useful for specifying non-standard coding practices".
   • Lulla et al. (T1) ran agents on 124 pull requests in 10 repositories with and without an AGENTS.md. With it, median runtime was 28.64% lower and output tokens 16.58% lower, "while maintaining a comparable task completion behavior".
   • The Agentic AI Foundation, which stewards the format, ran a small A/B test (July 2026): one repository, a twelve-line file, five runs per condition. On an ambiguous task, median wall time fell 27%, credits 24% and diff size 26%. On the harder, multi-file task, a single run per condition had shown the file making things 44% slower; five runs reversed that to a median gain of 9 to 10%.
   Measured by ETH Zurich and LogicStar researchers · benchmark and real-issue evaluation across several agents and models · February 2026, revised September 2026 · T1. Lulla et al., academic · paired runs with and without the file · 124 PRs, 10 repos · T1. AAIF · five runs per condition, one repository · July 2026 · T3 (the format's steward)

   The three agree once you separate what they measured. Success rates do not rise in general, and generated or generic files cost tokens. Short, specific, human-written files that state what the agent cannot infer make runs faster and cheaper without hurting completion. Anthropic's editing test: "For each line, ask: 'Would removing this cause Claude to make mistakes?' If not, cut it. Bloated CLAUDE.md files cause Claude to ignore your actual instructions!" Its include list is commands the agent can't guess, non-default style rules, test runners, repository etiquette, project-specific architectural decisions, environment quirks and non-obvious gotchas. Its exclude list includes "file-by-file descriptions of the codebase". GitHub's review of more than 2,500 agents.md files (vendor, qualitative) names six areas to cover (commands, testing, project structure, code style, git workflow and boundaries) and three tiers of boundary: "Always do", "Ask first" and "Never do". Many of the decisions that belong in the file are design decisions; the requirements, specs and design chapter covers how they reach the agent. The wiki compares the formats in AGENTS.md vs CLAUDE.md vs Cursor rules vs agent skills, and Appendix A, Templates, carries a starting file.

3. Attribute every agent pull request with a label, a trailer or the agent's own account. Without attribution none of the metrics in section 6.6 can be computed, and the phase cannot score above L1.
4. Name the task types you will delegate, and the ones you won't. GitHub's guidance for its own agent (vendor docs, read October 2026) is an exclusion list from the vendor with most to gain by saying yes. It keeps for humans "Broad-scoped, context-rich refactoring problems requiring cross-repository knowledge and testing", "Production-critical issues", "Tasks involving security, personally identifiable information, authentication repercussions", "Incident response" and "Tasks lacking clear definition". A delegated issue should carry "Complete acceptance criteria on what a good solution looks like".
5. Track success per task type from the first week, as merged divided by merged plus closed, and record whether a human had to push commits. A task type where humans routinely rescue the branch is L2 work, however high its merge rate.

You are at L2 when one named task type has attributed agent pull requests, all of them with recorded human review, an instruction file in the repository, and a success rate you can quote.

From L2 to L3: make the run safe to leave

1. Give the agent a check it can run, and gate the stop on it. Anthropic's guide describes an escalation. You can ask for the check in the prompt or set it as a session goal. You can make it "a deterministic gate": a Stop hook that "runs your check as a script and blocks the turn from ending until it passes". Or you can add a verification subagent so "the agent doing the work isn't the one grading it". The guide names the goal and the Stop hook as what let "an unattended run finish correctly without you", and only the hook is deterministic. Spotify's Honk puts its verifiers in exactly this position (case studies): if one fails, the pull request isn't opened. The testing and verification chapter covers making that check strong enough to trust.
2. Sandbox the run with both a filesystem and a network boundary. Anthropic's sandboxing write-up (vendor, October 2025) states it: "effective sandboxing requires both filesystem and network isolation". In its cloud sessions, "sensitive credentials (such as git credentials or signing keys) are never inside the sandbox"; a proxy checks each push and attaches the real token. Stripe's devboxes apply the same idea at company scale.
3. Take merge and approval out of the agent's reach. Copy the cloud-agent envelope: the agent pushes to its own branch only, workflows are held until a human reviews, and the requester cannot approve.
4. Pick the first L3 task type from your own data, by type rather than size (see the worked example). Prefer types whose definition of done is a check: a failing test that must pass, a lint rule, a dependency bump that must build.
5. Write the routing rule down and enforce it in code: which task types run unattended, who reviews what, and what sends a pull request back to full review. Then exercise the pause.
6. Start the 90-day comparison on day one. L3 requires escaped defects or incidents for the agent's work compared against a human baseline. dotnet's revert figures, 3 of 535 merged agent pull requests (0.6%) against 33 of 4,251 others (0.8%), are the only public agent-versus-non-agent revert comparison we found (the 4,251 include 666 bot pull requests), and the post says the difference is not statistically meaningful. Your sample will be smaller, so start counting early.

Worked example: task type versus task size

dotnet/runtime published success by category and by size for the same set of agent pull requests. Side by side, they show which axis to pick L3 candidates on.

dotnet/runtime agent PRs, May 2025 - Mar 2026 (success = merged / (merged + closed))

BY TASK TYPE                     BY SIZE (lines changed)
cleanup        84.7%  (77)       1-10         80.0%  (160)
testing        75.6%  (99)       11-50        76.9%  (195)
refactoring    69.7%  (86)       51-100       75.0%  (104)
bug fix        69.4% (317)       101-500      64.0%  (197)
feature        64.5% (131)       501-1,000    68.2%   (44)
performance    54.5%  (46)       1,001+       71.9%   (32)

spread by type:  84.7 - 54.5 = 30.2 points
spread by size:  80.0 - 64.0 = 16.0 points
(omitted: documentation, upgrades, "other"; 0-line PRs with no change)

Task type spreads success about twice as widely as size, and the size curve isn't even monotonic: the largest pull requests, over 1,000 lines, recover to 72%, "likely because these tend to be carefully scoped mechanical tasks". The post's summary is "task scope matters more than size." For this repository, cleanup and testing are the L3 candidates, and performance work stays at L2 however small the change. A 500-line mechanical cleanup is a better candidate than a 20-line performance tweak. These figures belong to one repository and one agent over ten months. Compute your own table before promoting anything, and expect the ranking, not the percentages, to transfer.

From L3 to L4: not yet

No public record shows auto-merged LLM-written code for a task class. If you try, the model's L4 conditions apply in full: an enumerated, reversible class; a documented verification record; continuous monitoring with automatic rollback; a named accountable owner; periodic audit samples. The narrowest honest candidate is a change class your deterministic tooling already auto-merges, with the LLM handling only the tail. That pattern belongs to maintenance and migrations.

Guardrails

Coding guardrails are mostly about where the agent runs and what it may approve, because those settings decide the level whatever the team intends. Rule 2 bites hardest here. An agent in a skip-permissions mode on a laptop holding production credentials is at L4 for whatever those credentials reach. The security, permissions and blast radius chapter owns the permission envelope per level. This is the subset a coding team configures.

• Lock permission modes centrally. Claude Code's permission docs (as of October 2026) say of bypassPermissions: "Only use this mode in isolated environments like containers or VMs where Claude Code can't cause damage". Administrators can disable bypass and auto modes in managed settings, "where they can't be overridden". OpenAI's Codex docs (as of October 2026) default to "no network access and write permissions limited to the active workspace". They mark the full-access flag --dangerously-bypass-approvals-and-sandbox "(not recommended)". Outside a sandbox, turn these modes off by policy, not convention.
• Don't treat approval prompts or allow-lists as a boundary. Anthropic's auto-mode write-up reports that users approve 93% of permission prompts. Its automated approver missed 17% of 52 curated real "overeager" actions, among them "attempting migrations against a production database". Claude Code's docs say a Bash deny rule "isn't a security boundary around the program".
• Put mandatory rules in hooks, not prose. "Unlike CLAUDE.md instructions which are advisory, hooks are deterministic and guarantee the action happens." "Never edit the migrations folder" belongs in a hook that blocks the write. See the wiki's lifecycle hooks and harness config.
• Keep the oracle out of reach. The agent must not be able to edit the tests, the CI configuration or the hook that judges it. That is the oracle principle, and the maturity model gives the evidence that agents game what they can reach. The book's proposal: require code-owner review from a human for test and CI files, and keep the agent's identity off that owner list.
• No production credentials where the agent runs. Stripe isolates devboxes from production and the internet, and Anthropic keeps git credentials outside the sandbox. Both put the boundary in the environment, where a model's mistake cannot cross it.

Metrics

Measure per task type, and pair every speed measure with one that would show it backfiring. Chapter 17, Measuring Success and When to Roll Back, owns the full set and the rollback triggers. These are the coding-specific ones.

• Success rate per task type: merged ÷ (merged + closed). This number decides promotion from L2 to L3.
• Unmodified-merge share: merged agent pull requests that went in without further commits (54.9% for Claude Code in open source). It separates "accepted" from "accepted after a human finished it".
• Human co-commit share: how often a human pushed to the agent's branch. A high rescue share means L2, whatever the merge rate.
• Review load: comments per merged agent pull request (16.5 against 12.4 in dotnet) and reviewer minutes. This is where coding agents' cost lands; the code review chapter covers keeping it in bounds.
• CI rounds per pull request. Stripe caps them at two. Rising rounds are an early sign the agent is iterating against the check rather than meeting it.
• Reverts and escaped defects against a human baseline: the stability pair for everything above, and the L3 entry requirement.
• Cost per merged change, including reviewer time, built as in the economics chapter.

Two rules on method. Never judge an instruction-file change, a model switch or a harness change on one run; in AAIF's test, five runs reversed what a single run had shown. And never report "% of code written by AI" or raw pull request counts as the outcome; Stripe's thousand a week shows scale and nothing else. The Thoughtworks Technology Radar (Vol. 34, April 2026) puts "coding throughput as a measure of productivity" in its Caution ring: "Cycle times increase as engineers raise PRs filled with insufficiently reviewed AI output". It proposes first-pass acceptance rate as a leading indicator. That is practitioner opinion, not measurement, but it points at the same thing as the unmodified-merge share.

Anti-patterns

Auto-approve on a workstation. Three public reports share one shape. In Gemini CLI issue #4586 (July 2025), a user reports losing their files after asking the CLI to organise a folder. The AI Incident Database's summary says the CLI acted on a failed directory-creation command as if it had succeeded, causing moves that "overwrote all but one file". In Claude Code issue #10077 (October 2025), a user reports an rm -rf that deleted their home directory. The reporter says the skip-permissions flag was not in use; that is questioned in the thread and unconfirmed. And according to TechRadar (secondary, December 2025), a Google Antigravity user with the auto-execute "Turbo" mode on lost an entire drive after asking for a project cache to be cleared. All three are single-user accounts with no vendor root cause, and the book draws no conclusion about any tool from them. The pattern is the point. An agent that may run destructive commands unseen, on a machine with no boundary, is at L4 for that machine. Nobody chose that level; a setting did.

The over-specified instruction file: long, generated by the agent itself, describing the codebase file by file. The ETH study found such files add cost without adding success. The Thoughtworks Radar lists "agent instruction bloat" under Caution: "As instructions grow, the likelihood increases that important rules are ignored… research suggests that hand-written versions are often more effective than LLM-generated ones." Prune it, and move anything mandatory into a hook.

Autonomy without scoping: handing an unattended agent real but loosely defined work and waiting. Answer.AI's verdict on what did work is the warning: "Tasks it can do are those that are so small and well-defined that I may as well do them myself, faster, my way." Name the task type first, and cap the run.

Judging the model before preparing the repository. A pilot that drops an agent into a repository whose build cannot reach its package feeds measures the firewall, not the agent. dotnet's first month, at 41.7%, would have ended many pilots; the same agent later ran at about 71%. Chapter 14, The 90-Day Pilot, makes environment preparation a precondition for any verdict.

Single-run evaluation. On AAIF's multi-file task, one run each way showed a 44% slowdown; five runs turned it into a median gain of 9 to 10%. Run-to-run variance is larger than most effects teams try to measure; the wiki's evaluating coding agents playbook covers the method.

Coding outrunning review. Promoting coding to L3 while review is still L1 turns unattended runs into an unread queue. If agents open pull requests faster than reviewers close them, invest next in code review and testing, not in more agents.