The 90-Day Pilot

What a pilot decides

"Should we adopt AI?" is not a question a pilot can answer. A pilot can answer a narrower one: should this task type, in this phase, move up one level? That is the unit the maturity model scores, and it is the unit a 90-day pilot can measure. The rest of this chapter follows from keeping the question that small. If you are running the pilot for a customer rather than inside your own team, Last Mile covers the other side of it: the terms that decide whether a pilot converts, and how to scope the first win.

Most of the published rollout record answers a different question: did people use the tool? Accenture's rollout, run with GitHub, is the best-documented example. 81.4% of developers installed the extension on the day they got a licence, 96% of those started accepting suggestions the same day, and 67% of survey respondents said they used it at least five days a week (GitHub with Accenture, May 2024, T3). ZoomInfo ran a four-phase rollout to more than 400 developers and reported an average acceptance rate of 33% for suggestions and 20% for lines of code, with 72% satisfaction (ZoomInfo, January 2025, T2). Neither number says whether anything shipped faster or broke less. ZoomInfo's paper gives no time-saved figure at all.

Uptake is not even guaranteed. In the three field experiments analysed by Cui et al., covering 4,867 developers, "around 30-40% of the engineers" did not even try a free, in-IDE assistant (Cui et al., February 2025 draft, T1, mixed independence; Economics, Risk and Measurement reads the effect sizes). So adoption is worth tracking, but as a precondition, not a result. A pilot judged on volume can look like a success while delivering nothing; the bottleneck chapter's pilot that looked like a success raised pull requests 13.6% and effective output 1%.

Weak claims are easy to spot. GitHub's Mercado Libre customer story gives "9,000+" developers in its header, "more than 10,000" in its body and 13,300 seats in its metadata, and the "~50%" reduction in time spent writing code is unattributed in the header; the body supports it only with one executive's account of spending "half as much time" and a reference to "the trials we've been running with our teams", with no design or results given (GitHub customer story, undated page, as of October 2026, T4). A pilot report that cannot state its own denominator should not decide a budget.

Pick the task: start in maintenance

Choose the pilot's task type from your assessment, not from a vendor demo. Assess Your Starting Point produces the input: each phase scored per task type, with its verifier, its credentials and its risk ceiling. From that list, a good first pilot passes four tests:

• A verifier already exists, and the agent cannot edit it. Build, type check, lint and tests decide pass or fail before any human looks. This is the oracle principle; without it the pilot measures reviewers, not the agent.
• A revert undoes it. Nothing the agent produces touches production state directly. The risk ceiling for the task must sit above the level you are testing.
• It recurs. Enough items arrive in 90 days to count. A one-off project gives you an anecdote.
• Someone owns it. A named team reviews the output and will live with the result.

Maintenance passes all four more often than any other phase, which is why this book recommends it as the first rung. Maintenance & Migrations shows why: the compiler and the existing tests were written before the agent arrived, they check behaviour the migration must preserve, and they do not care who wrote the diff. Pick one recurring change class there, such as a framework migration, a deprecated API or broken dependency bumps, and leave schema and data migrations out. A revert does not undo a dropped column.

If maintenance has no candidate, coding has some, but choose by task type, not size. In dotnet/runtime, agent pull requests for cleanup merged at 84.7% and for tests at 75.6%, against 54.5% for performance work (Microsoft .NET team, March 2026, T2; the full case is in Case Studies). Cleanup and test work are pilot candidates; performance work is not.

Be clear about the level under test. For a first pilot it is L2: the agent produces whole changes, and every one gets recorded human review. A 90-day pilot cannot certify L3, because the L3 definition requires an escaped-defect comparison over at least 90 days, after pause and rollback have been exercised. What a well-run L2 pilot does is collect the first 90 days of that comparison. It earns the right to attempt L3 for that task type; it does not grant it.

Prepare the environment before you judge the model

The most common way to waste a pilot is to drop an agent into a repository it cannot build, and then score the agent. In dotnet/runtime, the coding agent at first could not reach the package feeds its own build needed. The team opened the firewall to those feeds, wrote an instruction file and added setup steps to stop builds timing out. Pull requests created before those changes succeeded 38.1% of the time; those created after, 69%. The first month ran at 41.7%, and the most recent quarter at about 71%. The gain came "not through better AI models, but through better preparation" (Microsoft .NET team, T2).

A pilot that had ended after month one would have recorded 41.7% and stopped. So the book's proposal is to treat the first weeks as a shake-down and leave them out of the verdict. Run the agent where it will really run, watch where it fails, and fix the environment. Write a short instruction file by hand. Coding sets out the preparation steps and the mixed evidence on instruction files. When the failures that remain are about the work rather than the setup, freeze the configuration and start counting.

Preparation includes authority. Before the agent's first run, write down what the agent can reach: its credentials, its write scope, what it can merge, and whether any skip-permission mode is enabled. Then set it to the L2 column of the permission envelope: no production credentials, a human approving every commit, bypass modes disabled. A pilot run with a broader token is testing a level you did not choose.

Then check that your measurement can see the effect at all. When the Agentic AI Foundation, which stewards the AGENTS.md format, tested a twelve-line instruction file on a harder, multi-file task, a single run per condition showed the file making things 44% slower; five runs per condition gave a median gain of 9 to 10% (AAIF, July 2026, T3, one repository). Any configuration change made during the pilot, whether a model switch, a harness change or a new instruction file, needs repeated runs before anyone believes it. The wiki's eval variance and statistical power deep dive and its evaluating coding agents playbook cover the method.

If the task needs a written spec, count the spec as cost. A 14-engineer Spec Kit pilot put specification and planning overhead at 45 to 90 minutes per medium feature (Farrag, May 2026, T3, uncontrolled); Requirements, Specs & Design covers when that pays.

Take the baseline before the agent starts

A pilot without a comparison is a demo. The clearest published example of a controlled pilot before a scaled rollout is ANZ Bank's. Over six weeks in mid-2023 (two of preparation, four of Python challenges), it split more than 100 participants, drawn from roughly 5,000 engineers, into a group with Copilot and a control group. The Copilot group completed tasks "42.36% faster than the control group", a statistically significant result, and also showed fewer bugs and code smells; the security results were inconclusive (Chatterjee et al., T2).

The design is the lesson, not the number. The test behind the 42.36% had a sample size of 22, the tasks were exercises rather than the bank's own code, and the authors note that "participation rates fluctuated significantly across these tasks". ANZ then extended access to about 1,000 engineers and reported only unquantified "early data". It is still the right order: a controlled comparison first, a rollout second.

The common alternative is before and after, and it has a known flaw. The Spec Kit pilot compared two months before adoption with two months after: rollbacks fell from 2–4 a month to 0–1, and late-stage hotfixes per sprint from 3–5 to 1–2. Its author lists the limits: "the absence of a parallel control group", and "temporal confounds (e.g. improving AI model capabilities during the study period) cannot be excluded" (Farrag, T3). Models change within a quarter. A before-and-after pilot cannot separate your process change from the vendor's release notes.

The book's proposal for a pilot baseline has four parts:

• History. The previous 90 days of human-made changes in the same task type: count, lead time, review wait, reverts, defects traced back. The wiki's measuring ROI page puts the rule bluntly: no counterfactual baseline, no ROI.
• A concurrent comparison. Human-made changes in the same class during the same measured window, or a second team that does not get the agent. That removes the calendar confound that limited the Spec Kit pilot. The wiki's notes on online experiments for agents and shadow mode cover the designs.
• A pinned configuration. Fix the model, harness and instruction file for the measured window, and log any change.
• A cost line. Cost per merged change, including reviewer minutes and rework, from the model in Economics, Risk and Measurement. This book found no organisation that publishes this figure including human review, so the pilot is where you get your own.

Every flow metric is paired with a stability metric: lead time with reverts, landed changes with defects traced to them, review time with rework. The pairing is what stops a pilot repeating the volume trap.

Write the stop rule before the first result

A stop rule written after the results is a rationalisation. Write it first, name the person who applies it, and agree what happens when it fires. Two facts shape what the rule can be.

First, a 90-day pilot is too small to prove that a rare harm is absent. dotnet/runtime published the only public comparison of agent and non-agent revert rates this book found: 3 of 535 merged agent pull requests were reverted (0.6%), against 33 of 4,251 other merged pull requests, human and bot (0.8%), and the post says the difference is not statistically meaningful. A pilot will have far fewer changes. The arithmetic below shows what that means for a revert-based stop rule.

Illustrative arithmetic (no organisation). Binomial model.
Agent PRs merged during the measured window       150
Human revert rate in the same class (baseline)    0.8%
Expected reverts if the agent is no worse         1.2

Stop rule: "4 or more reverts in the window"
  P(rule fires) if agent revert rate = 0.8%       3%    (false alarm)
  P(rule fires) if agent revert rate = 3.0%       66%   (catches a ~4x harm)
  P(rule fires) if agent revert rate = 1.0%       6%    (misses a small harm)

A pilot-sized stop rule can catch a large harm and will miss a small one. So the rule should be an absolute threshold fixed in advance, not a significance test, and the small harms are left to the longer comparison that L3 entry requires. The wiki's page on online experiments for agents works through the sample-size arithmetic, and what to do when the experiment you need is unaffordable.

Second, early flow numbers may dip. DORA observed that "implementing a platform engineering initiative might lead to a temporary decrease in performance before improvements manifest" (DORA 2024, survey, October 2024). That finding is about platforms, not agents, but the shape is common to process changes, and dotnet's first month shows it. Stopping on a week-three slowdown kills pilots that would have worked.

The book's proposal, then, separates three kinds of trigger:

• Hard stops, immediate, on safety. The agent edits or skips its verifier; it acts outside its permission envelope; a change it produced reaches production and causes an incident above a severity you name. Any one ends the measured run and drops the task type back a level until the cause is fixed.
• Threshold stops, at review points, on stability. Reverts or traced defects above a count fixed in advance, as in the arithmetic above.
• Holds, at day 90, on flow and cost. No improvement in lead time or cost per merged change against the concurrent comparison. A hold means not climbing, not rolling back.

Measuring Success and When to Roll Back sets out the metric set and the published precedents for rollback triggers; the pilot's stop rule is the first instance of them.

The 90 days

The schedule below is the book's proposal. No published pilot follows it; it assembles the practices above in the order their dependencies impose.

The measured window is 45 days, not 90. That is deliberate: a measurement that includes the shake-down judges the firewall, and one that leaves no time to read the results produces a decision nobody can defend. If the task type's volume is too low to fill 45 days with enough changes to count, choose a busier task type rather than a longer pilot.

The scorecard is one page, signed before the agent's first run. Appendix A, Templates, carries the blank; the version below shows its shape.

PILOT SCORECARD (the book's proposal; sign before the first run)

Phase / task type     Maintenance / <one named change class>
Level under test      L2: every agent change human-reviewed, attributed
Verifier              build, type check, lint, tests; agent cannot edit
Credentials           own branch only; no production access; bypass off
Configuration         model, harness, instruction file: pinned on day 30
Baseline              previous 90 days, same class, human-made
Comparison            same class, same window, human-made
Flow metric           lead time; review wait vs active review time
Stability metric      reverts; defects traced within the window
Cost                  cost per merged change, incl. reviewer minutes
Hard stops            verifier edited or skipped; out-of-envelope action;
                      incident above severity <n>
Threshold stop        reverts >= <n> in the measured window
Hold                  no gain in lead time or cost vs comparison
Decider / date        <name> / day 90

Reading the result

A pilot has three honest outcomes, and each is a success if the rule was written in advance.

Extend. Flow improved against the comparison, stability held, and cost per merged change is known. Keep the task type at L2 and keep collecting the comparison until it covers the 90 days that L3 entry requires. Begin the L3 groundwork from Maintenance & Migrations: an evidence pack in place of line-by-line review, and pause and rollback exercised. Or pilot a second task type at L2. Do not do both at once.

Hold. Nothing broke, and nothing improved. Often the constraint is elsewhere: the agent's output waited in review, which is where the review-throttle arithmetic says it will wait. A hold points at the next pilot, which may belong in review or testing rather than generation.

Step down. A hard stop or a threshold stop fired. Return the task type to the level it came from, fix the cause, and decide whether to rerun. Under this book's model, stepping down is normal operations, not defeat.

Two limits apply to every outcome. The pilot's numbers belong to your codebase, your task type and this quarter's models; they do not transfer to other teams any more than Airbnb's 97% and Slack's 80% transfer to each other. And a pilot result is not a scaling result: ANZ's controlled six weeks were followed by a rollout to about 1,000 engineers that it reported only as "early data". What changes when the second hundred engineers arrive, from governance to permissions, is the subject of Scaling Across Teams.