Large-Scale Migration Agents

The bottleneck moved, and your tooling is still pointed at the old one.

Every instinct you have about a migration was formed when writing the change was the expensive part. That is why the traditional plan is "find someone who knows the codebase, give them six months" and why the traditional tooling is a codemod framework — something that makes writing the transformation cheaper. An agent inverts the economics completely. Generation drops toward zero and review does not move at all, because review is bounded by human attention, which does not scale with your budget.

Work the arithmetic once and the project plan writes itself. Ten thousand files at five minutes of careful human review each is 833 person-hours — five months of one engineer, spent entirely on reading changes an agent produced in a night. That is the actual cost of the migration, and no improvement in generation quality reduces it. Only two things do: raising the fraction of changes that a machine can approve, and reducing the number of changes a human must look at individually.

This is also why the first question to ask about a proposed migration is not "can the model do it" but "what would convince me it worked". If the honest answer is "an engineer reading it", the migration is not agent-shaped at this size, and the useful move is to shrink the scope until it is.

Build the oracle before you generate anything.

An oracle is any mechanical check that decides whether one migrated unit is acceptable. It is the single highest-leverage artefact in the project, and it should exist and be trusted before the first patch is written. In rough order of strength:

• Behavioural equivalence. Run both versions against the same inputs and compare outputs. The strongest available evidence, and the only one that catches a semantically wrong change that compiles and passes existing tests. Where you can record real inputs and replay them, do — a captured production trace makes a better oracle than any test you will write for this project.
• The existing test suite. Strong exactly where coverage is good, and quietly worthless where it is not. Measure coverage on the files you are migrating before you rely on it; migration targets are disproportionately old code, and old code is disproportionately untested.
• The compiler and the type checker. Cheap, fast, and only as informative as the type system. In a strongly-typed codebase this catches a large share of mechanical errors; in a dynamic one it catches almost nothing and should not be counted as verification.
• Property and invariant checks. Written for the migration itself: the public API must not change, no call site may gain a null, error handling must remain present. These are cheap to write, apply uniformly, and catch the specific ways this particular transformation goes wrong.
• A reviewing model. Useful as a filter that ranks and explains, not as an approver. See code review agents — a second model agreeing with the first is weak evidence, because both share the failure modes of the generation step.

Where coverage is thin, the correct first phase of the migration is usually to have the agent write tests against the current behaviour, land those, and only then migrate. That feels like a detour and is the opposite: characterisation tests are the oracle, they are independently valuable, and generating them is the task agents are best at. More on the mechanics in patch generation and tests.

Batch by verifiability, not by directory.

The default plan is to work through the codebase by package or by team, which optimises for organisational tidiness and produces a review queue of uniform difficulty — meaning uniformly slow. Sort by oracle strength instead, and the queue stratifies into three very different piles:

• Fully verified. Good coverage, clean compile, behavioural check passes. These merge on green with no individual human review — spot-check a random sample of 2%, not all of them. This should be the large majority of the work, and if it is not, you are missing an oracle.
• Partially verified. Compiles and type-checks, thin or absent tests. These get human review, but a narrowed one: the reviewer is answering "is this semantically the same", not re-reviewing the whole file, and the diff should be presented with the specific unverified property called out.
• Unverifiable. No tests, dynamic dispatch, reflection, generated code, a file nobody owns. Do not migrate these with an agent. Route them to a human, or leave them and record the debt explicitly.

Doing the sort first is what makes the project schedulable, because the three piles have completely different throughputs and you can only forecast a migration once you know their sizes. It also surfaces the unpleasant fact early: a codebase where the third pile is 40% of files has a testing problem that this migration will not fix and should not try to.

Write the codemod for the head; give the agent the tail.

The most common design error is to hand the entire corpus to the agent because the agent can handle it. It can, and it will do so at a per-file cost thousands of times higher than a deterministic transform, with variance a deterministic transform does not have. Real migrations are power-law distributed: a small number of mechanical patterns cover most call sites, and the remainder are all different.

Split accordingly:

• An AST-based codemod handles the head. Deterministic, reviewable once rather than per file, free to re-run, and correct by construction. If 70% of your call sites are the same rename, that 70% should never reach a model. Compiler-aware tooling in this space exists precisely for this, and the token savings are large enough to change the project budget.
• The agent handles the tail. The call sites with unusual control flow, the ones where the migration requires understanding what the code is for, the ones where a mechanical transform would be syntactically valid and semantically wrong. This is genuinely where a model earns its cost, and it is a much smaller corpus than the whole.
• The agent also writes the codemod. Have it study a sample, propose the transformation, and then apply the deterministic version everywhere. This is the highest-leverage use of a model in the whole project: one careful generation, ten thousand deterministic applications.

Two practical notes. Give the agent per-file context rather than repository context — the constraint everyone hits is that neither a human nor a model holds a large codebase in working memory, so the winning pattern is a narrow, precisely-assembled context per unit of work plus tools to look things up on demand. See repo navigation and context and context engineering. And run each unit in a fresh sandbox with the build and test harness available, so the agent can iterate against the oracle itself rather than guessing — an agent that can run the tests fixes most of its own mistakes before you see them. See sandboxing and execution.

Running the fleet without melting anything.

Once units are independent, the work is embarrassingly parallel, and the limits you hit are operational rather than intellectual:

• Provider throughput, not wall-clock. A thousand concurrent migration agents is a token-rate problem long before it is a CPU problem, and the failure arrives as sustained throttling. Meter your own concurrency rather than discovering the ceiling; see concurrency and scaling.
• CI capacity. Every unit runs a build and a test suite, often several times as the agent iterates. Migration runs routinely cost more in CI minutes than in tokens, and a build farm sized for human commit rates will be the actual bottleneck. Measure this before you launch the fleet, not after.
• Cost per landed change, not per token. An agent that burns fifteen attempts and succeeds is often cheaper than one that fails cleanly and hands you a manual task. Track spend against changes that merged, the way agent cost control argues, and cap per-unit attempts so one pathological file cannot consume the budget.
• Failure is a signal, not just a retry. Cluster the failures before re-running them. Twenty files failing the same way is one missing rule in your codemod, and fixing it is worth more than twenty retries. See tool error recovery.

Ownership is the part that is easy to get wrong socially. A pull request authored by an agent still needs a human accountable for it, and "the migration team" is not an owner that survives an incident six months later. Attribute each change to the team that owns the file, land it through their normal process, and make the migration's own dashboard show landed-versus-open per team — otherwise the last 15% never lands, which is the standard way these projects end.

When not to do this.

Some migrations are a bad fit, and recognising them early is worth more than any technique above:

• No oracle is available and none can be built. If correctness lives entirely in a human's head — a UI refactor judged on feel, a change to code whose behaviour nobody can characterise — an agent produces volume you cannot check. Scope down to the part that is checkable.
• The transformation is genuinely mechanical. If a codemod does the whole job, use the codemod. A model is a worse tool for a task with a deterministic solution: slower, more expensive, and with a non-zero error rate on work that has a zero-error alternative.
• The migration is really a redesign. "Move to the new framework" often smuggles in a set of architecture decisions. Those need to be made once, by people, and written down — then the agent applies them. An agent asked to decide the design ten thousand times will make ten thousand slightly different decisions, and the inconsistency is worse than the old code.
• The code should be deleted. Migrations are a good moment to discover how much of a codebase is dead. Check usage before you spend anything migrating a file, and measure the project on the outcome rather than on files touched — see measuring ROI.

Related: evaluating coding agents for measuring the generator itself, coding agent architecture for the loop underneath all of this, and eval variance and statistical power for why your pilot needs more than one run before you believe its number.