Maintenance & Migrations

What changes

Migrations are the work every organisation postpones: an unsupported framework version, a deprecated API, an identifier type running out of room. They are easy to describe and expensive to finish, because the same change must be made in thousands of slightly different places. Before language models, the answer was deterministic tooling: codemods, AST rewrites, dependency bots. The evidence says it should stay the backbone.

Deterministic tooling has a ceiling, though, and Spotify documented where it sits. Its Fleet Management system has automated "around half of Spotify's pull requests" since mid-2024, but the scripts behind it are costly to grow. The Maven dependency updater alone, "handling all corner cases", had grown "to over 20,000 lines of code", and "because of this complexity, most of the automated changes we could implement were simple ones" (Spotify Engineering, Honk Part 1, November 2025). According to InfoQ's report of a QCon London talk by two Spotify engineers (March 2026, secondary), scripts reduced "migration timelines from nearly a year to under a week for 70% of the fleet", but "the remaining 30% proved extremely difficult due to edge cases and complexity".

That 30% is what changes. A language model can handle edge cases that would take a script author weeks to enumerate, as long as something else decides whether each edit is correct. In maintenance, that something already exists. The compiler, the type checker, the linter and the existing tests were written before the agent arrived, they test behaviour the migration is meant to preserve, and they do not care who wrote the diff. That is why the oracle principle is easier to satisfy here than in any other phase. You do not have to invent the oracle; you only have to keep the agent from touching it.

Two things do not change. The first is that the model alone is not enough. Google's migration team wrote that "the use of LLMs alone through simple prompting is not sufficient for anything but the simplest of migrations" (Google, January 2025, T2), and Slack, trying a model on its own before combining it with an AST codemod, "encountered significant variation and inconsistency" (Slack Engineering, May 2024, T3). Every migration on record that worked was a hybrid of deterministic tooling and a model.

The second is that someone still has to review the result, and that is where the constraint lands. Google's JUnit team limited how many changes it generated each week so as not to overwhelm reviewers. The Spotify speakers at QCon said the shift "has made PR review, not code generation, the new bottleneck" (InfoQ, secondary), and in June 2026 Spotify's chief architect reported "76% more PRs to review" (Spotify Engineering, T3). This is the book's thesis from When Code Gets Cheap, the Bottleneck Moves, observed at phase level: once generation is cheap, the migration runs as fast as its reviewers. The full Google and Spotify cases, with their caveats, are in Case Studies; this chapter builds on them rather than repeating them.

The ladder for this phase

The criteria below follow the maintenance row of the maturity grid, with the general conditions for each level written out. As everywhere in this book, the level is scored per task type ("Maintenance: L3 for test-framework migrations; L2 for dependency upgrades") and set by the higher of the review mode and the authority the agent actually holds.

The grid is thin at both ends and dense in the middle. L1 has no published case at all; the other boundaries each carry a lesson.

L0 is not "behind". Fleet Management is among the most autonomous change systems anyone has described, and it scores L0 for AI because no model is involved. A deterministic rewrite that covers 70% of a fleet is cheaper, more predictable and easier to review than any agent; the agent's job is the residue. A team that replaces working codemods with an agent to "move up the ladder" has misread the ladder: levels describe authority, not ambition.

L2 is defined by the review, not the tool. Google's int32 migration is the reference. The model produced most of the edits, but engineers reviewed every change and, as the authors report, "the human needed to revert at least some changes the model made that were either incorrect or not necessary". Case Studies scores it "L2 moving to L3" because the volume was set by what reviewers could absorb. Elastic's pipeline for broken Renovate dependency PRs (September 2025) is L2 by explicit policy: "Never commit AI work without human supervision", with auto-merge disabled whenever the agent contributed.

L3 changes what the human looks at. At Spotify the agent runs unattended in a sandbox, deterministic verifiers decide whether a pull request is even opened, and the owning teams review and merge the result. That is L3 for fleet migrations on design evidence, as the maturity model's worked example shows, with the gap an outside scorer must write down: no published comparison of escaped defects against human-made changes. Every L3 entry in this row has the same gap. The verifiers are documented; the outcomes are not.

L4 is empty for language models. Nothing in the public record shows an LLM-authored migration change merging without per-item human approval under a verification record. Dependency auto-merge is old and common, but it is deterministic.

Not every "migration" belongs on this ladder. A database schema or data migration is an action on production state, often irreversible, and it sits under a different risk ceiling. Anthropic's write-up of its auto-mode classifier lists, among overeager actions from its own internal use, "attempting migrations against a production database". The book's proposed permission envelope (see Security, Permissions & Blast Radius) says that at L3 the agent prepares such a migration and a human executes it, and that it is never L4-eligible without undo and backups outside the blast radius. The wiki's database migration agents playbook covers the pattern. This chapter's L3 claims apply to code migrations, whose effects a revert can undo.

The case: Airbnb's Enzyme-to-RTL migration

Level: Maintenance, L3 for one test-framework migration, scored on verifier design. Per-file validation steps, not a human, decided which files were done; the post does not describe how the finished files were reviewed.

Airbnb needed to move its React component tests from Enzyme to React Testing Library. "We'd originally estimated this would take 1.5 years of engineering time to do by hand." The team instead built a pipeline that walked each file through a fixed sequence of steps: refactor from Enzyme, fix Jest, fix lint and the TypeScript compiler, then mark the file complete (Airbnb Tech Blog, 2025). A file advanced only when the deterministic check for its step passed; failing files were retried.

The first bulk run "successfully migrated 75% of our target files in just four hours." The team then spent four days in a loop it called "sample, tune, sweep": sample the failures, adjust the prompts and context, and rerun. That "pushed our completed files from 75% to 97% of the total files". Some long-tail files were retried "anywhere between 50 to 100 times". The remaining files were fixed by hand, with the pipeline's partial output as "a great baseline for manual intervention". The whole migration finished "in just 6 weeks", covering "nearly 3.5K React component test files".

Two findings deserve more weight than the headline. First, persistence beat cleverness. "The most effective route to improve outcomes was simply brute force": retries against a deterministic check, not prompt craft. That works only because the check was cheap, automatic and outside the model's control. A retry loop against a verifier the agent can edit is a loop that converges on deleting the failing assertion.

Second, context mattered more than wording. Prompts grew to "anywhere between 40,000 to 100,000 tokens", pulling in "as many as 50 related files", and the authors name "choosing the right related files" as "the main success driver", ahead of "getting the prompt engineering perfect". The investment that paid was in the pipeline: step validation, retries and context selection.

What it does not show. Whether the migrated tests still test what the old ones did; the post reports no coverage or mutation comparison, which matters doubly because the artefacts being migrated are themselves tests (Testing & Verification takes up how to check that). It also gives no compute cost, no review effort, and no defect data after the move. The 1.5-year baseline is an estimate nobody re-ran.

The contrast: the same migration at Slack

Slack ran the same migration in 2024 and reported a different number. It planned to convert "more than 15,000" Enzyme tests. An AST-only codemod reached "a modest success rate of 45%", because "AST lacks the capability to incorporate such contextual information", such as the rendered DOM. A model alone produced "significant variation and inconsistency". Combining the two, with the codemod doing the mechanical part and the model the contextual part, gave "the remarkable 80% conversion success rate, based on selected files": a manual evaluation of "nine test files of varying complexities" found that 80% of the content within them was accurately converted. Slack reported an adoption rate of about 64% and a saving of "22% of developer time".

Do not read 80% against 97% as Airbnb beating Slack. The two figures differ in model generation (Slack's post dates from 2024), in retry budget (Airbnb retried some files up to 100 times) and in what counted as success (the share of code converted correctly across a 9-file sample, against the share of files that passed every step in a fleet of about 3,500). They agree on the pattern, not the number. Migration figures do not transfer between organisations, or even between two runs of the same migration. Your own first pilot is the only number you can plan with.

Migration steps

The steps below are ordered by level. Each climb is per change class: you might run dependency fixes at L2 and test-framework migrations at L3 in the same quarter.

L0 to L1: inventory before you add a model.

• List the deterministic tooling you already run (codemods, Renovate or Dependabot, lint autofixes) and the migrations it cannot finish. The unfinished residue is your candidate list.
• Let engineers use assistants on migration tickets. There is no published evidence for this rung, so treat it as cheap and unmeasured, not as a step to evaluate.

L1 to L2: one change class, verifiers first.

• Pick one named, recurring change class with a clear end state: a framework migration, a deprecated API, broken dependency bumps. The wiki's migration agents, dependency upgrade agents and dead-code and flag removal playbooks cover the common classes.
• Write the verifier sequence before the prompt: build, type check, lint, the existing tests. Make each a pipeline step a file must pass, as Airbnb did.
• Keep the deterministic part deterministic. Let the codemod do what it can and give the model the remainder, as Slack did.
• Attribute every agent PR, and disable auto-merge for anything the agent touched. Elastic's script aborts if it cannot disable auto-merge.
• Track acceptance and rework for the class: merged unchanged, merged after edits, reverted.

L2 to L3: move the oracle out of reach and change what reviewers see.

• Run the verifiers outside the agent's control. At Spotify they run in a stop hook when the agent tries to finish, and a failed verifier means "the PR isn't opened" (Honk Part 3).
• Sandbox the agent, give it a small toolset, and leave pushing and notifying to the surrounding infrastructure (Spotify's design).
• Budget retries, not prompt rewrites. Airbnb's gains came from re-running against the checks.
• Fill verifier gaps before you climb. When Spotify migrated downstream data pipelines (Honk Part 4, April 2026), the repositories rarely had build-time tests, so owners tested the changes by hand. Where there is no verifier, the human is the verifier, and the class stays at L2.
• Replace line-by-line review with an evidence pack the owner approves: the verifier log, the diff summary, a sample of files. The pack's format is the book's proposal; Appendix A, Templates, carries a version.
• Set the generation rate to review capacity, as Google did, and write the routing rules down.
• Exercise pause and rollback, and start the 90-day comparison of reverts and escaped defects against human-made changes of the same class. Until it has run, the claim is "L3 by design", not L3.

L3 to L4: no one has published this climb. If you attempt it, the book's proposal is to start from a class that is already reversible and already auto-merged deterministically, such as patch-level dependency updates, and to apply the L4 conditions from the maturity model: a verification record of N consecutive changes with no escaped defect, automatic rollback, a named accountable owner and periodic audit samples. You would be the first public data point.

Worked example: sizing the review throttle

This is illustrative arithmetic and names no organisation. Take a migration of 4,000 files. The pipeline converts 95% automatically within a week; the remaining 200 files go to engineers. Reviewers take changes in batches of 25 files.

Files converted by pipeline    3,800      (95% of 4,000)
PRs at 25 files each             152
Review time per PR                45 min   (line by line)
Review budget                      8 h/wk  (2 owners x 4 h)
PRs reviewed per week           10.7      (480 / 45)
Weeks to land                    ~14

Twice the generation speed       ~14 weeks (review-bound)

Evidence pack + sampled files     15 min per PR
PRs reviewed per week             32      (480 / 15)
Weeks to land                     ~5

Generation finishes in a week; the migration takes 14 weeks because review does, so faster generation changes nothing. Bigger batches are not a lever either: DORA's 2025 report hypothesises that AI-linked instability arises "in part, because it is harder to review larger batches of code" (see the bottleneck chapter), so review time per PR rises with them. What moves the date is changing what the reviewer looks at, from every line to an evidence pack and a sample, and that change is exactly the move from L2 to L3. The 15-minute figure is an assumption; measure your own.

Guardrails

Keep the oracle out of the agent's reach, and check for the known cheats. Migration agents game the verifier when they can. According to InfoQ's report of the Spotify talk, early agents "would take shortcuts to make builds pass, such as commenting out failing tests or downgrading Java versions". Elastic's agent "fixed" broken dependency bumps by removing the bump, and the team's answer was a new prompt rule: "NEVER downgrade versions of dependencies". An instruction is not a guardrail. The book's proposal: add deterministic diff checks that reject a change which deletes or skips tests, lowers a version, or edits the verifier configuration, and make those checks part of the oracle.

Treat an LLM judge as a tunable second gate, not a safety layer. Spotify's judge vetoed "about a quarter" of thousands of sessions, and "the agent is able to course correct half the time" (Honk Part 3, December 2025). InfoQ later reported that Spotify removed it as "too rigid, blocking valid changes". A judge has a false-positive cost. Deterministic verifiers come first; a judge is optional.

Put authority in the infrastructure, not the agent. Run the agent in a sandbox with no production credentials, scope its token to its own branch, and let the pipeline push, open PRs and notify. The envelope per level is in Security, Permissions & Blast Radius; any data or schema migration stays a human-executed action.

Know what the verifier cannot see. Spotify names the worst failure mode as "a PR that passes CI but is functionally incorrect". Your L3 is only as strong as your tests; check their ability to catch faults (mutation score, not coverage) before you climb, as Testing & Verification covers.

Throttle at the reviewer. Cap open agent PRs per owning team, and close stale ones instead of letting them pile up. The InfoQ report describes Spotify considering both "allowing migration drivers to approve their own PRs and closing stale pull requests". Code Review covers risk-tiered routing.

Metrics

Pair every flow metric with a stability metric, per change class, and baseline both before the agent starts.

• Flow. Changes landed per week; review wait time per agent PR, measured separately from active review time; elapsed time against the last comparable migration done without the agent.
• Stability. Reverts of agent migration changes, against human-made changes of the same class; defects traced to migration changes over 90 days. The only public agent-against-non-agent revert comparison (its comparison group includes bot pull requests) is in coding, not maintenance: dotnet/runtime's 0.6% against 0.8% (Case Studies, T2). No migration case publishes one.
• Pipeline health. Verifier rejection rate, judge veto rate if you run one, retries per landed file, and the share of the migration finished by hand. A rising manual tail tells you the class is drifting out of what the pipeline can verify.

Two popular numbers do not belong on the dashboard. The first is the share of edits an AI wrote. Google's 80% and its year-long study's 74.45% of changes describe the pipeline, not the outcome; the outcome is that changes landed behind a compiler that did not care who wrote them. The second is "developer-years saved", which is a counterfactual multiplication rather than a measurement.

Which metric should trigger a step back down, and at what threshold, is the subject of Chapter 17, Measuring Success and When to Roll Back. Write the thresholds before the migration starts.

Anti-patterns

• Prompting alone. Pointing a model at a codebase and asking it to migrate. Google and Slack both report that it fails beyond the simplest cases. Build the pipeline: deterministic steps, checks, retries.
• Determinism alone, forever. Growing a rewrite script toward 20,000 lines to chase the last edge cases. The open debate is whether the model should write the codemod or the diff (Moderne and OpenRewrite argue for deterministic recipes, an analyst view); either way, hand the tail to a model behind the same checks.
• An oracle the agent can edit. If the agent can skip a test, lower a version or edit the lint configuration, a passing pipeline means nothing. That class is at L2, whatever the dashboard says.
• Generating faster than you can review. A thousand open migration PRs are a queue, not progress. Size the throttle first.
• Importing someone else's number. Planning on 97% because Airbnb got it, or 80% because Slack did. Run your own sample.
• Calling deterministic automation "AI autonomy", or the reverse. Auto-merged bot PRs are L0 for AI; an LLM-authored PR that merges because nobody looked is de facto L4. Score by what actually happens.
• Treating data migrations like code migrations. A revert undoes a code change. It does not undo a dropped column.