Case Studies: Who Did It and What They Measured

How to read a case

The same number means different things depending on who counted it and what they counted. "73% accepted" from a filtered test generator, "75% useful" from an opt-in rating button and "79% shipped unchanged" from a CEO's post are three different measurements that happen to look alike. This chapter prints every case in the same format so you can tell them apart:

• A tier badge in the case header, from T1 to T4.
• The level, tagged by granted authority. The level is what the AI's credentials and settings actually allowed, read against what a human reviewed, not what the team called the deployment. This is Rule 2 of the maturity model, and it matters most in the failures.
• A "Measured by" line directly under the case: who measured, by what method, over what n or window, and the resulting tier.
• What it does not show. Every case has a hole. Naming it keeps one case from being stretched to cover a claim it never tested.

The tiers, in four lines (the Preface gives the full legend). A source's tier is the lower of who measured and how they measured.

• T1, independent measurement. A party with no product stake, with a stated method and denominators.
• T2, organisation-measured with a stated method. The organisation running the system measured it, with denominators, often peer-reviewed.
• T3, telemetry or report with thin method. Engineering blogs, pooled vendor telemetry and uncontrolled pilots: numbers without a full method, baseline or n.
• T4, claim. Executive statements, press relays, marketing and disputed accounts, with no method.

Why these ten. They are the public cases that publish a method or a failure. Six are positive or mixed and four are negative. Several famous cases are left out on purpose, and the reasons are the chapter's argument in miniature:

• Stripe reports over a thousand merged pull requests a week produced entirely by its unattended agents, every one reviewed by a human (Stripe, as of February 2026, T3). It publishes no success or failure rate, so the case shows volume, not outcome. It belongs to Chapter 6, Coding.
• Airbnb moved nearly 3,500 test files in six weeks against an estimate of 1.5 years by hand (Airbnb Tech Blog, T3). The baseline is an estimate nobody ran. It belongs to Chapter 11, Maintenance & Migrations.
• Amazon's "4,500 developer-years" is the most-quoted figure in the field. It returns at the end of this chapter as a reading exercise.

The ten at a glance:

Where it worked: verification closed the loop

The six cases in this section succeeded to different degrees, and they share a mechanism. In each one, something deterministic decided whether the AI's output was acceptable before a human spent attention on it: a compiler, a test suite, a coverage delta, a build. Where that check was strong, the AI was allowed to do more. Where it was missing, the humans did the checking, and the humans became the bottleneck.

Google: LLM-assisted migrations · T2 · organisation-measured

Level: Maintenance, L2 moving to L3 (see the maintenance row of the grid). Engineers reviewed the changes, and the volume was set by review capacity.

Google's experience report on LLM-driven code migrations (January 2025) covers several large, mechanical changes. In the int32-to-int64 identifier migration in Google Ads, about 80% of the code modifications in the landed changes were fully AI-authored, and the engineers estimated that total migration time fell by about 50%, review included. The JUnit3-to-JUnit4 migration changed 5,359 files and more than 149,000 lines in three months, and about 87% of the AI-generated code was committed without any change. For Joda-Time to java.time, the team extrapolates a saving of about 89% from small clusters.

The paper is as useful for its caveats as for its numbers. The pipeline combined the model with Google's existing code-search and indexing tooling, and the authors say plainly that "the use of LLMs alone through simple prompting is not sufficient for anything but the simplest of migrations." In the int32 migration, the authors report, "the human needed to revert at least some changes the model made that were either incorrect or not necessary." The JUnit team limited how many changes it generated per week so as not to overwhelm reviewers. Generation was never the constraint; review was.

A follow-up year-long study (FSE 2025 industry track) tracked 39 migrations run by three developers over twelve months: 595 code changes and 93,574 edits, of which 74.45% of changes and 69.46% of edits were LLM-generated. The developers again estimated a reduction of about 50% in total migration time against earlier manual work. The share of edits an LLM produced is a description of the pipeline, not a success metric. The outcome evidence is that the changes landed, behind a compiler and test suite that did not care who wrote them.

What it does not show. Quality over time. The authors write that "whether there is a long-term impact on quality remains to be seen." The time savings are estimates against manual efforts nobody re-ran. The wiki's playbook on migration agents covers the pipeline pattern; Chapter 11, Maintenance & Migrations, builds on this case.

Meta: TestGen-LLM and ACH · T2 · organisation-measured

Level: Testing, L2 (see the testing row). The model generates; deterministic filters discard; an engineer accepts or rejects each survivor.

TestGen-LLM (FSE 2024 industry track) extends existing Kotlin test classes. On Instagram's Reels and Stories products, "75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage." Only candidates that clear all three filters reach an engineer. In Meta's test-a-thons the tool improved 11.5% of the classes it was applied to, and "73% of its recommendations were accepted for production deployment by Meta software engineers."

Read those two numbers together. A quarter of the raw output was worth anything; nearly three-quarters of what survived the filters was accepted. The difference between 25% and 73% is the work done by the build, the repeated test runs and the coverage measurement, none of which the model could influence. That is the oracle principle in its plainest form: the filter, not the model, decides what is correct.

ACH, Meta's mutation-guided generator, uses deliberately injected faults as targets. Run on 10,795 Android Kotlin classes across seven platforms, it produced 9,095 mutants and 571 privacy-hardening tests. Engineers accepted 73% of the tests and judged 36% privacy-relevant. Meta is careful not to oversell the mechanism: "Mutation testing cannot exist on its own (it requires a test to already exist)", and the company names the test-oracle problem as the open challenge (Meta Engineering, September 2025).

What it does not show. Acceptance measures whether an engineer agreed, not whether a test catches the faults that matter. Neither paper reports a fall in escaped defects. When Meta later tried to generate tests that catch bugs in a change before it lands, precision became the problem; Chapter 8, Testing & Verification, takes that up. For the tooling, see the wiki's test-generation playbook.

Spotify: Honk · T3 · engineering-blog telemetry

Level: Maintenance, L3 for fleet-wide migrations. The agent runs unattended in a sandbox, deterministic verifiers gate every change, and the owning teams review and merge the resulting pull requests.

Spotify's four-part series is the most detailed public description of a background coding agent's design. Part 1 reports "more than 1,500 pull requests that teams across Spotify have merged" (as of November 2025), and, for the migration types it lists, "a total time saving of 60–90% compared to writing the code by hand". The post gives no baseline method for that range. The authors add: "We don't have all the answers yet."

The design choices are what transfer:

• Verifiers first. Formatting, build and test verifiers are selected automatically from a repository's contents and run when the agent tries to finish. If one fails, "the PR isn't opened and the user is presented with an error message" (Part 3).
• A sandbox, and authority kept outside the agent. "It's highly sandboxed." Pushing the branch and posting to Slack are done by the surrounding infrastructure, not by the agent.
• A deliberately small toolset. The agent gets a verify tool, limited Git and Bash with a strict allow-list, and no code search. "The more tools you have, the more dimensions of unpredictability you introduce" (Part 2).
• An LLM judge as a second gate. Out of thousands of sessions, "the judge vetoes about a quarter of them", and "the agent is able to course correct half the time."

The judge's story did not end there. According to InfoQ's report of a QCon London talk (March 2026, secondary), Spotify later removed the judge as "too rigid, blocking valid changes", and the same talk described agents "commenting out failing tests or downgrading Java versions" (InfoQ). No Spotify primary repeats either point. Read together, they say an LLM judge is a tunable gate with a false-positive cost, and that agents game what they can reach. Deterministic verifiers come first.

Two later posts sharpen the lesson. Part 4 (April 2026) rolled out 240 automated migration pull requests to downstream data pipelines, but those repositories rarely had build-time tests, so owners had to test the changes by hand. Where the verifier was missing, the human became the verifier. And in June 2026 Spotify's chief architect reported pull-request frequency up 76% (Spotify Engineering): "we now have 76% more PRs to review." Spotify titled its write-up of the talk "Coding Is No Longer the Constraint".

What it does not show. Failure or revert rates for merged agent changes, and the exact denominator behind "a quarter". Spotify's separate Fleet Management system has merged more than 2.5 million automated maintenance pull requests, "the vast majority auto-merged with no human in the loop" (Spotify, as of June 2026), but that system is largely deterministic. In this book's terms that is L0 for AI, however autonomous it looks.

Microsoft: a coding agent in dotnet/runtime · T2 · organisation-measured

Level: Coding, L2 (see the coding row). Every agent pull request was human-reviewed. Cleanup and test work are the candidates for L3, as the maturity model's worked example scores it.

Stephen Toub's "Ten Months with Copilot Coding Agent in dotnet/runtime" (March 2026) is unusual because it publishes the numbers that do not flatter the tool. Over 878 agent pull requests between 19 May 2025 and 22 March 2026, the agent's success rate (merged as a share of merged plus closed) was 67.9%, against 87.1% for Microsoft's human engineers, 79.7% for community contributors and 85.9% for bots. The post calls the populations different and the comparison directional.

The headline lesson is in a before-and-after split. Pull requests created before the team's first setup changes succeeded 38.1% of the time; those created after succeeded 69%. The gain came "not through better AI models, but through better preparation": firewall rules for package feeds, a copilot-instructions.md file revised many times, and setup steps to avoid build timeouts. Task type spread the results more widely than size did. Cleanup succeeded 84.7% of the time and testing 75.6%, while performance work succeeded 54.5% of the time; by size, the rate ranged only from 64% to 80%. Pull requests where a human pushed commits succeeded 86.2% of the time, against 55.1% without.

The post also counts what the agent cost reviewers. Merged agent pull requests averaged 16.5 comments, against 12.4 for merged human ones; the count includes every comment on the pull request, the agent's own replies and bot messages among them. And it reports the only public revert comparison between agent and non-agent pull requests found for this book (the non-agent group includes 666 bot pull requests): 3 of 535 merged agent pull requests were reverted (0.6%), against 33 of 4,251 other merged pull requests (0.8%). The post says "the difference is not statistically meaningful, but at minimum, the data shows no red flags."

What it does not show. Cost. The post "does not analyze the compute cost of CCA usage or the CI resources consumed by CCA PRs", and it adds, "These are real costs." It also cannot separate the effect of the instruction file from the environment fixes that shipped with it, so it is not evidence that instruction files alone raise success. Chapter 6, Coding, builds its preparation checklist on this case.

Uber: uReview · T3 · engineering-blog telemetry

Level: Review, L2 moving to L3 (see the review row). AI is the first-pass reviewer on almost every change; humans still approve every merge.

uReview (August 2025) "analyzes over 90% of the weekly ~65,000 diffs" landed at Uber. That is Uber's headline figure; the same post elsewhere says "65,000 per month", so the inconsistency is in the primary. Engineers who interact with the tool mark 75% of its comments useful, and more than 65% of its posted comments are addressed in the same changeset, by an automated check. Uber sets this against internal audits showing that only 51% of human-written comments "are considered as bugs by the author" and addressed in the same changeset. The two figures come from different methods, a model re-run for the AI comments and author judgement for the human ones, so the comparison is Uber's and is not like for like.

The result is engineered, not lucky. Three assistants run in parallel (bugs, best practices and application security). A confidence score with per-category thresholds filters their comments, near-duplicates are merged, and whole categories that developers did not value, such as readability and style, are suppressed. A separate grader model scores the generator's output. Uber heads the lesson "Precision Is More Valuable than Volume" and writes that "comment quality matters far more than quantity." An AI reviewer that comments on everything trains people to ignore it; one that stays silent unless it is confident gets acted on.

What it does not show. Whether "addressed" comments were correct, or whether defects fell. The reviewer sees only the code, not "past PRs, feature flag configurations, database schemas, technical documentation, and so on", so design review stays with humans. The estimated saving of about 1,500 hours a week assumes ten minutes of human review per commit, a counterfactual rather than a measurement. Chapter 7, Code Review, covers how to compare review metrics; the wiki's code-review agents playbook covers the tooling.

Google: DIDACT build repair · T2 · randomised controlled trial

Level: CI/CD, L1 (see the CI/CD row). The model suggests a fix for a broken build; the developer decides whether to apply it.

This is the smallest effect in the chapter and the most trustworthy. In a post published in April 2024, Google reports that it "randomly assigned 50% of all Google developers to the treatment group" for eleven weeks, giving them ML-suggested fixes for build breakages in the IDE. "About 34% of users experiencing a build breakage in a given month end up applying such ML-suggested fixes." The measured result was a "~2% increase in CL throughput". Google also looked for harm: "We also detect no observable increase in safety risks or bugs", with no detectable difference in rollback rates or sanitizer failures.

Two things make this case matter beyond its size. It is the only randomised trial in this book's evidence that measured downstream safety, not just speed. And its 2% is what an honest effect looks like when half of all developers are in the treatment group, the other half form the control, and the task is narrow. It measures one task class and should not be set against the coding-speed trials that Economics, Risk and Measurement reviews.

What it does not show. Anything about agents that push fixes themselves. DIDACT is L1: the human applies every fix. Chapter 9, CI/CD & Release, covers the agentic versions; the wiki's CI repair agents playbook covers the tooling.

Where it did not: two negative results

Negative results are rare in this literature, and they are worth more per word than positive ones. They are the cases you can calibrate a pilot against. Both of these come from parties with no product stake.

An industrial AI-review study · T1 · independent measurement

Level: Review, L2. An AI reviewer commented on every pull request, and humans approved merges.

An ICSE 2025 industry-track study deployed a reviewer built on the open-source Qodo PR-Agent at an industry partner, across three projects and 1,568 pull requests. By the usual vendor metric it did well: 73.8% of the AI's comments were resolved. But mean pull-request closure time rose from 5 hours 52 minutes to 8 hours 20 minutes, a statistically significant increase overall. The trend varied by project: closure time rose in two of the three and fell in the third. Developers named the drawbacks as "faulty reviews, unnecessary corrections, and irrelevant comments."

The case explains why Uber's precision filters matter. A reviewer that generates more comments adds work for the author and the human reviewer, even when many of the comments are resolved. "Resolved" counts activity. Closure time counts what the team waited for.

What it does not show. Whether a reviewer tuned for precision would have done better at the same site, or whether code quality changed. It is one partner, one tool configuration and one period.

Answer.AI's month with Devin · T1 (n=20) · independent practitioners

Level: Coding, an attempted L3. Real, loosely scoped tasks were handed to an autonomous agent to finish on its own.

In "Thoughts On A Month With Devin" (January 2025), three Answer.AI practitioners gave Devin 20 real tasks. Three succeeded, fourteen failed and three were inconclusive. The failures had a shape. The agent "would spend days pursuing impossible solutions rather than recognizing fundamental blockers", and where it did produce code, one author found "spaghetti code that was way more confusing to read through than if I'd written it from scratch." Their summary of what worked: "Tasks it can do are those that are so small and well-defined that I may as well do them myself, faster, my way."

The tool has moved on since early 2025, so do not read this as a product review. The point is about scoping. Without a narrow task type and a check the agent must pass, autonomy turns into persistence on the wrong problem. Compare dotnet/runtime: there, task type predicted success, and preparation nearly doubled it.

What it does not show. How the current version of the tool would score, or how a scoped, verifier-gated deployment of the same agent would do.

Where it broke: authority without a boundary

The two failures below are the best-documented incidents of their kind. Both are T4: neither has an independent forensic account. They are here because the facts that matter, what the agent could reach and what changed afterwards, are acknowledged or primary-sourced on both sides. Read them for the boundary that was missing, not for the model that was used.

Replit: the production database · T4 · acknowledged by vendor and user

Level: Coding and operations, de facto L4. The agent held write access to a production database. A code freeze had been declared, but a freeze stated in conversation is an instruction, not a boundary.

In July 2025, during an experiment by the SaaStr founder Jason Lemkin, who was building an app with Replit's agent and documenting it on X, the agent deleted the production database during a code freeze. The AI Incident Database records the incident date as 18 July 2025 and states that the agent "incorrectly claimed rollback was impossible, delaying recovery." Replit's CEO, Amjad Masad, responded publicly (The Register, 22 July 2025): "Unacceptable and should never be possible." He said the agent "didn't have access to the proper internal docs", and that Replit had "started rolling out automatic DB dev/prod separation to prevent this categorically". He also pointed to the existing one-click restore of a project's state, and said staging environments were "in the works".

The fix is now documented in Replit's own product documentation (Replit Docs, as of October 2026). On the development database, the agent "can create and modify tables". On production, the agent "can't touch it": "Agent is not able to modify the production database." Schema changes reach production only when the app is published.

That is the cleanest exhibit in the public record of an environment boundary as the fix. Replit did not respond by asking for a better model or more careful prompts. It removed the agent's ability to reach production. Before the fix, whatever the user thought they had delegated, the agent's credentials put that database at L4. After it, the agent cannot reach production at all, and its changes arrive there only through publishing.

What it does not show. The exact record counts, which circulate only in secondary press and are not used here. It also does not show how often such deletions happen. No published incident rate by permission level exists.

Amazon Kiro and AWS Cost Explorer · T4 · disputed

Level: Operations, de facto L4 for the affected environment before the incident; rolled back to L2 after it. Whatever review mode was intended, a role broad enough to delete and recreate a production environment set the level.

On 19 February 2026 the Financial Times reported that an AI coding tool had been involved in an AWS outage the previous December. The FT article is paywalled, and its account is known here only through Reuters and Engadget. Amazon published a rebuttal the next day, "Correcting the Financial Times report about AWS, Kiro, and AI", and gave an expanded statement to the press. The two accounts, side by side:

In its statement to the press, quoted by Engadget (updated 21 February 2026), Amazon added that the staff member had "broader permissions than expected — a user access control issue, not an AI autonomy issue." Those words are not in the blog post.

This book does not adjudicate the dispute, and it does not attribute the incident to "the AI" or to "the human". What matters is where the two accounts meet. Both put access scope at the centre. The FT describes an agent that could delete and recreate an environment, and Amazon blames a misconfigured role. Amazon's own explanation is therefore a statement about granted authority. And Amazon's own remedy, mandatory peer review for production access, is a step down the ladder from de facto L4 to L2. It is the clearest primary-sourced example of an organisation stepping operations authority back to human-gated after an incident. Chapter 17, Measuring Success and When to Roll Back, builds its rollback triggers from cases like this one, and Chapter 12, Security, Permissions & Blast Radius, carries the fuller incident set.

What it does not show. The duration (13 hours or "brief") and Kiro's causal role are both contested. The case should never be quoted as "an AI took down AWS". It should be quoted as "an over-scoped role took down a service, and the fix was a review gate on production access." See the wiki's blast radius concept for the general principle.

The pattern across ten cases

Lay the ten side by side and five patterns hold across tiers, companies and phases.

The first and fourth patterns are the two halves of the chapter's argument, and they are the same idea seen from opposite sides. A verifier the agent cannot edit bounds what a wrong output can do before it lands. An environment boundary the agent cannot cross bounds what a wrong action can do after it is taken. The three cases that went furthest toward L3 all had the first. Both failures lacked the second. The other three patterns explain why the successes were smaller than their headlines. Review absorbed the extra output, preparation did more work than model choice, and once someone ran a control group the effect was modest.

Notice also what the tiers do to the story. The strongest evidence (T1 and the T2 RCT) is either negative or small. The most impressive numbers (Spotify's time savings, Uber's hours saved) are T3, and the most dramatic events are T4. That is not a reason to dismiss the T3 and T4 cases. It is a reason to take their mechanisms, the verifiers, the sandboxes and the boundaries, and not their magnitudes.

What the record does not contain

The gaps are as important as the cases, because they mark where any plan you make is running ahead of public evidence.

• No large-company L3 or L4 rollout measured on outcomes. No organisation has published change-failure rates, incident rates or escaped defects for an agent deployment at L3 or above. Stripe and Spotify publish no revert rates, and Google says only that reverts happened. The dotnet/runtime revert comparison (0.6% against 0.8%) is the only public agent-versus-non-agent comparison of its kind (its comparison group includes bot pull requests), and its own author says the samples are too small for the difference to be statistically meaningful.
• No published rollback trigger for coding agents. No company has published criteria for rolling back an AI coding tool or reducing an agent's autonomy. Amazon's peer-review response to Kiro is a rollback, but not a stated trigger. The nearest precedents come from elsewhere. Google's Tricorder platform tracks developers' "Not useful" clicks on each static-analysis check and disables any check whose ratio goes above 10% until its authors improve it (Google, CACM 2018, T2; it predates LLMs). Google's SRE guidance describes operations agents that "automatically downgrade the request to L2" when they detect an elevated risk score (T3). Chapter 17, Measuring Success and When to Roll Back, turns these into the book's proposed triggers, labelled as the book's proposal.
• No cost per merged change that includes human review. The dotnet post declines to count compute and CI costs, and no case here reports reviewer hours per merged agent change. Economics, Risk and Measurement shows how to build that number for yourself.

Worked example: re-reading a headline. The most-quoted figure in AI-assisted maintenance comes from Amazon's CEO. Put it through the five questions this chapter asks of every case (they are listed in the box at the end).

Run the questions in order:

1. Who measured? Answered: Amazon, about its own product. That fixes the tier but does not disqualify the claim.
2. Against what baseline? Fails. "Typically 50 developer-days" is an estimate of what manual upgrades would have cost, not a measured control.
3. Over what window and n? Fails. "Tens of thousands" of applications and "under six months" for "more than 50%" of the production Java systems, but no exact count and no window for the developer-years figure.
4. What does "success" count? Fails as an outcome. "Shipped without additional changes" measures reviewer acceptance, not correctness. Like Meta's 73%, it says nothing about defects found later.
5. What authority did the AI hold? Fails. The posts do not say what the agent could change or how its output was gated.

One question answered, four failed. None of that means the upgrades did not happen or did not save real effort. A maintenance task behind a compiler is exactly where this chapter's evidence says L3 can work. It means the number cannot carry an argument. "4,500 developer-years" is a counterfactual estimate whose method Amazon has not disclosed, and it belongs in a vendor box, not a business case. Economics, Risk and Measurement takes the arithmetic apart. Here the point is the habit: ask the five questions before you quote any number, including your own team's.