Economics, Risk and Measurement

The range, and what moves it

Four randomised studies anchor the debate. Their results are about 75 percentage points apart, yet read together they describe one mechanism, not four verdicts.

The spread is not noise. The gains come from greenfield or standardised work, often done by less-experienced developers: Peng's participants built a new server from nothing, and both Peng and Cui found that less-experienced developers gained most. The loss comes from experts working on large codebases they already knew intimately, where checking the AI's output most plausibly ate the gain. Google's enterprise task sits between them. A smaller bank experiment points the same way: ANZ's Copilot group finished tasks 42.36% faster, but the timing test had a sample of 22 and used self-reported times (ANZ Bank, T2). So the effect depends on four things. Treat them as the moderators of any estimate you make:

• Task type. New code against a clear spec gains; changes to existing behaviour gain less.
• Familiarity. The better a developer already knows the code, the less the assistant adds and the more its errors cost.
• Codebase maturity. Large, old, high-context repositories leave the least room.
• Tool generation. Each study measured the tools of its year. A result from one generation does not transfer automatically to the next.

The largest dataset that splits results by these moderators is not a trial. The Stanford Software Engineering Productivity group scored the git history of more than 100,000 engineers and published its own "Orientative Guidelines" (Denisov-Blanch, slides, T3). Greenfield, low-complexity work gains +35–40%; greenfield, high-complexity work +10–15%; brownfield, low-complexity +15–20%; and brownfield, high-complexity work only +0–10%. A second matrix in the same deck, split by language popularity, puts high-complexity tasks in niche languages such as COBOL, Haskell or Elixir at −5% to +5%. A slide title in the same deck gives slightly different figures for the same cells (30–35%, 10–15%, 15–20%, 5–10%), one more reason to quote ranges and the authors' own label, never a single figure.

Most enterprise work is brownfield. If yours is, plan from the bottom-right of that matrix, not the top-left. The jagged frontier describes the same pattern at the level of individual tasks.

Why clean trials are getting rare. METR tried to rerun its study from August 2025 with 57 developers and more than 800 tasks. "30% to 50% of developers" said they had skipped submitting tasks they did not want to do without AI, so the tasks that stayed in the experiment were not a fair sample. METR's point estimates leaned towards a speed-up, with wide intervals (−18% time for returning developers, CI −38% to +9%; −4% for new ones, CI −15% to +9%). METR's own verdict is that the data give "an unreliable signal" (METR, February 2026, T1). That is not a reversal of the 2025 result. Developers now refuse to work without the tools, which says something about perceived value and nothing precise about measured value. You will rarely get a fresh external trial that fits your codebase; the credible evidence left is your own before-and-after data, with enough runs and a long enough window to see past the noise (see eval variance and statistical power).

Perception is not measurement

The most abundant number about AI productivity, what people say it did for them, is also the least reliable. METR's developers forecast a 24% speed-up before the study. After working through their tasks, they still believed AI had made them about 20% faster. The stopwatch said 19% slower. That is a sign error, not a rounding error, made by experienced engineers about their own work. The Stanford group found the same thing at a different scale: across 43 engineers, self-rated productivity correlated with measured productivity at r = 0.17, and people misjudged their own productivity by about 30 percentile points (slides, T3, small n). Meanwhile DORA's 2025 survey of 4,867 respondents found that 90% use AI at work and more than 80% believe it has increased their productivity (DORA 2025, survey).

Organisation-level numbers sit far lower than that belief. Sundar Pichai said Google's company-wide "engineering velocity" gain from AI was "now at 10%", while describing it as a "tough measure" and giving no definition (Lex Fridman Podcast transcript, June 2025, T4). Bain reports that "Teams using AI assistants see 10% to 15% productivity boosts" (Bain, 2025, T3; the page does not say what the figure rests on). The largest telemetry series, from DX, is lower again (see the box). Beyond software teams, Danish administrative data on the wider labour market found "precise null effects on earnings and recorded hours", "ruling out effects larger than 2% two years after the launch of ChatGPT" (Humlum & Vestergaard, NBER, T1). That is a labour-market result, not a delivery result, but it is a reminder that individual time savings need not show up in aggregate outcomes.

The gap between how fast people feel and how much more ships is the subject of fractional time savings: minutes saved inside a task rarely become whole tasks delivered. It is also why the maturity model's rules refuse self-placement without observable evidence. For your business case, the rule is short. Self-report can tell you whether people like a tool. It cannot tell you whether to pay for it.

Volume is not value: "% of code written by AI"

The most quoted AI number in executive decks, the share of code that AI wrote, is undefined, unauditable, and not a measure of anything a customer receives.

Follow one company's series and watch the definition move. In October 2024 Google said (Alphabet Q3 2024 earnings call, T4) "more than a quarter of all new code at Google is generated by AI, then reviewed and accepted by engineers." In April 2025: "25% of code that's checked in involves people accepting AI-suggested solutions … That number is well over 30% now" (Alphabet Q1 2025 earnings call, T4). In April 2026: "75% of all new code at Google is now AI-generated and approved by engineers, up from 50% last fall" (Google Cloud Next 2026, T4). "Generated by AI", "involves people accepting AI-suggested solutions" and "AI-generated and approved" are three different numerators. None of the statements says how a line counts as AI's.

The same CEO drew the distinction himself. In June 2025, having just mentioned "around 30% of code now uses AI-generated suggestions or whatever it is", Pichai called engineering velocity "the most important metric", put it at 10%, and said it was "different than … 'This percentage of code is now written by AI'" (podcast transcript, T4). Google has published no updated velocity figure (as of October 2026). Other firms show the same looseness:

• Microsoft. Satya Nadella, as reported by The Register: "maybe 20 to 30 percent of the code that is inside of our repos today in some of our projects are probably all written by software" (April 2025, T4, secondary). Note the hedges: "maybe", "some of our projects", "probably".
• Coinbase. Its head of platform said "close to 100% of our code, probably somewhere between 95% and 100%, is written by or with LLMs today", while core cryptography is still written mainly by humans and checked line by line by senior cryptographers (Cointelegraph, July 2026, T4). "By or with" covers everything from autocomplete to unattended agents.
• DX's quarterly reports. AI-authored code was 22% (Q4 2025), then 27.4% (Q1 2026), then 52.7% (Q2 2026), all T3. The 27.4% is defined as code "merged without major human rewrites or modifications" and measured "by asking developers directly". The Q2 release compares its 52.7% with "24% two quarters ago", not the 22% DX published at the time, and press coverage reports that DX rebuilt its method in between. Three numbers, at least two methods: this is not a series.

Even with a fixed definition, the share counts input, not outcome. It rises when developers accept more suggestions, whether or not anything ships faster or breaks less. The bottleneck chapter shows a pilot where pull requests rose 13.6% while effective output rose 1% and rework rose 2.6×. A code-share metric would have scored that pilot a success. Two rules follow. Never put "% of code by AI" in a business case or a success criterion. And never compare one company's figure with another's: they are not measuring the same thing.

Counterfactuals dressed as measurements

The biggest "savings" figures in the public record are not measurements. They are estimates of work that never happened, multiplied by a count, and reported in the past tense.

Do the arithmetic and you can see how the headline is built. This is illustrative arithmetic, assuming about 250 working days in a developer-year. 4,500 developer-years is about 1,125,000 developer-days. Divided by the 50 developer-days a manual upgrade "typically" takes, that is about 22,500 applications, which is consistent with the "tens of thousands" the AWS post mentions. So the headline is essentially a count of upgraded applications multiplied by an estimated manual cost. The "few hours" per upgrade was not subtracted in any published way, and neither was the time engineers spent on the 21% of generated code reviews that needed changes. The only ratio in the claim that sounds measured, 79%, comes with no denominator. Run the five questions from the case-studies chapter (who measured, against what baseline, over what window and n, what counts as success, what authority the AI held) and the claim fails four of them.

Better-documented cases share the same structure, to a lesser degree:

• Airbnb migrated nearly 3,500 React test files in six weeks, against a manual estimate of a year and a half (Airbnb Tech Blog, T3). The six weeks were measured. The year and a half was an estimate.
• Uber reports "approximately 1,500 hours saved weekly, equivalent to nearly 39 developer years annually" for its uReview bot. The inputs are an internal benchmark that a second human reviewer would need 10 minutes per commit, and "over 10,000 commits" a week (Uber Engineering, T3). The post does not show the multiplication; 10,000 × 10 minutes would be about 1,670 hours, so the published 1,500 has been adjusted in some unstated way. The commits were counted. The 10 minutes saved per commit is an estimate.
• McKinsey published a 2025 survey of nearly 300 senior leaders, 100 of whom assessed impact. It reports that top performers saw "16 to 30 percent improvements" in productivity, customer experience and time to market, and 31–45% in software quality (McKinsey). The outcomes were self-assessed by the same respondents, and "top performers" were the top quintile on those same outcomes. The comparison is circular, so part of the gap it reports is built in by definition. As outcome evidence, it is T4.

A counterfactual still tells you where an organisation believes the value is. But it belongs in the "hypothesis" column of your plan, not the "evidence" column: ask which part was measured and which part was multiplied. The wiki's measuring ROI page covers how to build the baseline these claims lack.

What it costs

Seats to tokens. The flat seat price that made AI coding look cheap is ending, and the reason is autonomy. GitHub moved Copilot from premium requests to token-based "AI Credits" on 1 June 2026. Its stated reason: "Agentic usage is becoming the default, and it brings significantly higher compute and inference demands." Usage is now "calculated based on token consumption, including input, output, and cached tokens", and "Copilot code review will also consume GitHub Actions minutes" (GitHub changelog, April 2026). Seat prices stayed the same, $19 per user for Business and $39 for Enterprise, each now including an equal dollar value of credits (as of April 2026). Above the allowance, cost scales with how much work the agents do. The more you delegate, the more you pay. The wiki's pricing models page compares the options.

Gartner forecasts that "By 2028, AI coding costs will overtake the average developer's salary due to rising large language model (LLM) token consumption and the shift to consumption-based licensing models", and its analyst adds that "Most organizations still lack the maturity and frameworks to effectively measure cost versus business impact" (Gartner, June 2026). That is a forecast, not a measurement; the second sentence is the more useful one.

Review is the line that does not shrink. Token prices may fall. Human review minutes do not fall with them, and agent output creates more of them. In dotnet/runtime, merged agent pull requests averaged 16.5 comments each against 12.4 for human ones, counting every comment type, including the agent's own replies and bot messages (Microsoft .NET team, T2). The same team was candid about what its ten-month study left out: it "does not analyze the compute cost of CCA usage or the CI resources consumed by CCA PRs". It added, "These are real costs". AI review moves some of that work onto tokens, but a human still approves the merge. The wiki's cost of human review page explains why this line dominates once generation is cheap, and the bottleneck chapter shows where the queue forms.

Budgets without outcomes. Usage pricing without per-team budgets fails fast. According to press reports citing The Information, Uber exhausted its full-year 2026 AI budget by about April, driven by Claude Code. The reports say adoption had risen from 32% to 84% of engineers; per-engineer spending figures differ between write-ups and none traces to an Uber source, so this book does not repeat them. Per-engineer token limits reportedly followed. Its COO said it was hard to link more tokens to more useful features (Quartz, 2026).

DX's pooled data show the same shape across 500+ organisations. Median quarterly AI spend went from roughly $1.5K to $44K in a year ("nearly 28x" in the tech sector; the unit is not stated), while DX's "innovation ratio" held "at roughly 57–58%" (DX, July 2026, T3). Take the direction, not the magnitude: spend rose by more than an order of magnitude and the outcome DX tracks did not move.

Worked example: cost per merged change. No organisation has published a cost per merged agent change that includes human review time. The .NET team explicitly left compute out. The one public cost-to-merge experiment, Insight's, left human review out. So you will have to compute your own. The formula is simple:

cost per merged change =
    ( agent tokens
    + CI minutes x CI rate
    + AI review
    + reviewer minutes x loaded rate
    + rework )
  / merged changes

Published inputs give you a starting range. In Insight's experiment, one real feature (the RFC 8628 device flow) was built 29 times across 23 model-and-harness combinations, gated by an AI reviewer and a 38-test oracle. The cost to reach a mergeable state ran from $2.59 to $33.38 and took from 26 minutes to about three hours. On the same model, the harness alone moved the cost by roughly 2.5×. Cost was computed from tokens at published rates and the reviewer was a model, so no human review time is in the figure (Insight, July 2026, T3, one feature). For the denominator, merge rates are well below 100%: of the 878 agent pull requests opened in dotnet/runtime, 67.9% of those that were merged or closed were merged, against 87.1% for Microsoft's own engineers.

Here is the formula filled in. This is illustrative arithmetic, not data from any organisation: token cost and AI review are set inside the published ranges, and CI cost, reviewer time, loaded rate and rework are assumptions you should replace with your own.

Per 100 agent PRs attempted                    USD
  agent tokens    100 x $20                  2,000
  CI              100 x $3                     300
  AI review       100 x $20                  2,000
  human review    100 x 30 min at $100/h     5,000
  rework          10 follow-ups x $93          930
                                            ------
  total                                     10,230
  merged (67.9% of 100)                         68
  cost per merged change                      $150

Same, token cost halved (better harness):     $134
Same, human review minutes halved:            $110

On these assumptions, human review is about half the total, and halving review time saves more than twice as much as halving token spend. Your numbers will differ; that is the point. Until you know which line dominates in your own pipeline, you do not know whether to invest in a cheaper model, a better harness or fewer, smaller pull requests. Chapter 14, The 90-Day Pilot, uses this model as its cost baseline.

What it risks

Costs show up on an invoice; risks do not, so they rarely reach the business case. Three kinds are worth pricing now. Chapter 12, Security, Permissions & Blast Radius, covers them in depth, including the authority risk behind every destructive agent incident on record.

The code itself. The strongest agent-era number is independent. On 186 real feature requests in open-source repositories, an agent setting built on SWE-Agent and Claude 4 Sonnet produced code that worked 57% of the time but was secure only 11.8% of the time (SusVibes, ICML 2026, T1). Vendor data agree on direction. Veracode's 2026 leaderboard says the security pass rate has "stalled at 56%" while syntax passes almost always. Reasoning models scored 56% against 51% for others; large, medium and small models scored 53%, 51% and 51%; coding-specialised models did no better than general ones (Veracode, July 2026, T3). In other words, roughly 44% of security-relevant outputs still fail, and neither a bigger model nor a coding-tuned one fixes it.

The reviewer is part of the risk. In a controlled study at Stanford, participants with an AI assistant "wrote significantly less secure code" and were "more likely to believe they wrote secure code" (Perry et al., T1, 47 participants, using codex-davinci-002). An author's confidence is therefore not evidence of safety. That is why security scanning belongs in the pipeline as a gate, not in the reviewer's head.

The supply chain. Models invent package names. Across 576,000 samples from 16 models, the hallucinated-package rate was "at least 5.2% for commercial models and 21.7% for open-source models" (Spracklen et al., USENIX Security 2025, T1). The more dangerous finding is that the inventions repeat: "43% of hallucinated packages were repeated in all 10 queries". An attacker can predict them and register them. A 2026 single-author replication on five newer models found lower rates, 4.62% to 6.10%, but 127 names that all five models invented identically (Churilov, not peer-reviewed). A dependency allow-list costs less than one incident.

The legal position (as of October 2026; details in Appendix B, Regulatory and Licensing Status):

• On 16 September 2026, in Doe v. GitHub, the Ninth Circuit rejected the DMCA §1202(b) "output" theory. Copilot and Codex do not "remove or alter" copyright-management information; they "instead create new works that never contained that information". Two breach-of-contract claims "remain pending" (opinion).
• The US Copyright Office's January 2025 report says that prompts alone do not give a user enough control to be the author of AI output (copyright.gov; wording via secondary summaries). For a codebase that is mostly agent-written, ownership is an open question for counsel.
• The EU AI Act's Annex III, point 4(b), treats as high-risk any system used "to monitor and evaluate the performance and behaviour of persons" in work relationships (Annex III), with application moved to 2 December 2027. An AI dashboard that scores individual developers may fall under it. That is the book's inference, pending legal review, not a settled reading.

The wiki's IP and copyright for agent output page tracks the detail.

Price these risks as line items, not footnotes: static analysis on every agent change, secret scanning, a dependency allow-list, and the review hours they generate. The wiki's cost of agent errors page shows how to size them.

A policy for reading vendor numbers

Most numbers you are handed are true in a narrow sense and useless in the sense you need. The fix is a reading policy, applied the same way every time, including to the vendors you like.

1. Tier every number. A source's tier is the lower of who measured and how. T1 is independent measurement with a stated method; T2 is an organisation measuring its own system, with denominators; T3 is telemetry or a report with a thin method; T4 is a claim. A famous name does not lift a figure out of T4.
2. Box what a vendor says about its own product. Learn from it, but never let it carry the argument, whoever the vendor is.
3. Accept direction, not magnitude, from pooled telemetry. Faros, LinearB and Jellyfish agree that AI brings more and larger pull requests. They disagree on whether review time rises or falls, possibly because waiting time and active time move in opposite directions (the bottleneck chapter reconciles them). Believe the sign where they agree; do not plan on the percentage.
4. One framing per report, and no splicing. Faros's 2026 report gives median time in review as +441.5% in one place and "5x" in another (Faros, April 2026, T3). Pick one. Its next report gave incidents per pull request as "up just 14.5%", after +242.7% in the previous report (Faros, September 2026, T3). Those are two reports with different windows and an unstated baseline, not a trend. Never combine figures from different vendors, windows or definitions into one number.
5. Find the denominator and the definition. "Accepted", "useful", "addressed" and "resolved" are different metrics, often computed by a model. Chapter 7, Code Review, gives a vocabulary for them. If you cannot find what a percentage is a percentage of, discard it.
6. Prefer sources that publish their bad numbers. The .NET team's disclosure that its agent's merge rate was 67.9% against 87.1% for Microsoft's own engineers is a number you can calibrate against, precisely because a vendor had reasons not to publish it.

The wiki's reading benchmarks page applies the same discipline to model evaluations. The agent pullback is a measurement failure argues that disappointing deployments usually lacked it from the start. Chapter 17, Measuring Success and When to Roll Back, turns the policy into a metric set: a flow metric paired with a stability metric for each phase, and rollback triggers written before the pilot starts.