Code Review

What changes

Review is where the bottleneck lands. When Code Gets Cheap, the Bottleneck Moves set out the telemetry: more pull requests, bigger pull requests, longer waits, and a growing share merged with no review at all. This chapter takes that queue as given and asks what AI can do about it.

There are two different answers. The first makes AI a commenter: a bot that reads every diff and posts findings. That adds work, because every comment has to be read and then fixed or dismissed. The second makes AI part of a router: the system decides which changes need a person, what that person must look at, and which can pass with an audit sample. That removes work. The first can backfire. In the industrial study that Case Studies reads in full, 73.8% of the bot's comments were resolved, and mean pull-request closure time still rose from 5 hours 52 minutes to 8 hours 20 minutes (ICSE-SEIP 2025, T1).

The human side changes too, in two ways.

Confidence stops being a signal. In a controlled study at Stanford, participants with an AI assistant were "significantly more likely to write an insecure solution" on an encryption task, and also more likely to believe they had written secure code (Perry et al.).

An author's certainty was never a good proxy for care, and with an assistant in the loop it gets worse. Review has to rest on checks and evidence, not on how sure the author sounds.

Capacity is finite, and inbound volume can exceed it. curl ended its bug bounty at the end of January 2026. Its maintainer, Daniel Stenberg, said the aim was to "remove the incentive for people to submit crap and non-well researched reports to us. AI generated or not", and that "The current torrent of submissions put a high load on the curl security team" (The Register). That is a security inbox, not a pull-request queue, but the mechanism is the same: when output costs less to generate than to review, reviewers get buried. curl did not review faster. It cut the inflow.

So the question for this phase is not "should a bot review our pull requests?" It is: which changes still need a person, what does that person look at, and what evidence lets everything else through?

The ladder for this phase

The criteria below are the ones in the review row of the maturity grid. As everywhere in this book, the level is what a human reviews and when, read against what the AI is allowed to do. For review the authority question is unusually sharp: does any AI hold the right to approve?

L1 sits on the author's side. Google's system proposes an edit that resolves a comment a human reviewer already wrote. AI touches the response to review, not the review itself.

L2 is where almost every product sits today, by design. A bot reads the diff, posts comments, and a human decides. Anthropic's Code Review, for example, "won't approve PRs — that's still a human call" (Anthropic, vendor blog, as of March 2026). GitHub's Copilot code review, on "more than one in five code reviews on GitHub", is described by the feedback it posts, not the merges it allows (GitHub, as of March 2026). Because a human approves every merge, these tools are L2 however good their comments are. The useful L2 question is whether the bot reduces human effort or adds to it.

L3 changes the reviewer's object. The human stops reading every diff and reads only what the rules route to them, plus a structured summary of evidence for the rest. Hassan et al. call that summary a Merge-Readiness Pack, one of the "structured evidence bundles submitted for human approval, designed to demonstrate functional completeness" along with sound verification, engineering hygiene, rationale and auditability (Hassan et al., conceptual). The evidence is partial: Atlassian's fall in human-written comments shows that what humans review can change, not that the change was safe. No organisation has published the comparison the maturity model's L3 criteria require: the escaped-defect rate of the AI-routed changes against a human baseline over at least 90 days (see the L3 definition).

A reverse signal points the other way. According to the FT, as reported by Fortune in March 2026, Amazon's retail organisation planned senior sign-off for AI-assisted changes by junior and mid-level engineers after a run of incidents. Amazon said only one incident discussed was AI-related, none involved AI-written code, and no such sign-off is required (T4, disputed). Either way, the response under discussion was more human review, not less.

L4 has a precedent but no LLM evidence. Zalando's bot, in the case below, approves a defined low-risk class without a human. The post never says whether a model is involved, and a rule-based approver is L0 for AI, so the cell stays empty for LLMs. Zalando is a precedent for the routing mechanism, not AI evidence.

One heuristic from the maturity model's rules applies here directly: run Coding at most one level above the lower of Review and Testing. That is the book's proposal, not a law. If coding agents open unattended pull requests (Coding L3) while review is still L1, expect the queue. Raise review first; Coding shows the other side of the trade.

The case: Atlassian's reviewer, and Zalando's router

The argument has two halves, precision and routing, and the best evidence for each comes from a different company. Atlassian measured an AI commenter carefully. Zalando published a router built from incident data.

Atlassian: RovoDev Code Reviewer · T2 · organisation-measured

Level: Review, L2. The reviewer comments on pull requests; humans approve every merge.

Atlassian's engineers evaluated their RovoDev Code Reviewer in production and published it as an ICSE 2026 industry-track paper (arXiv 2601.01129, January 2026). The abstract reports three figures from offline, online and user-feedback evaluations "over a one-year period". 38.70% of the reviewer's comments "triggered code changes in the subsequent commits". Pull-request cycle time fell 30.8%, which the current (v2) abstract presents more cautiously, as "the promise of accelerating feedback cycles". And the number of human-written review comments fell 35.6%.

The first figure is the kind vendors lead with, and the least informative: a comment that "triggered code changes" may have caught a bug or prompted an unneeded rename. The second is a flow measure, and it points the opposite way from the industrial study, where closure got slower. Tool, site and period differ, so the two cannot be reconciled from outside. Together they show that an AI reviewer's effect on flow is not fixed.

The third figure is the hardest to read. A 35.6% fall in human-written comments is what L3 looks like from outside: what humans review changed. But it fits two stories. In one, the bot caught the routine issues and humans spent their attention elsewhere. In the other, humans deferred to the bot and looked less closely. Only an escaped-defect comparison separates them, and the paper does not report one.

What it does not show. Whether production defects rose or fell, how the 30.8% splits between waiting and active review, or whether the result transfers. Peer review constrains the method, not the independence: Atlassian measured its own tool.

Zalando: risk-based auto-approval · T3 · engineering blog

Level: Review, a routing mechanism at the L3/L4 boundary, with no stated AI component.

In "Agentic Engineering at Zalando: a snapshot" (August 2026), Zalando describes a bot that sorts every pull request into rollout risk classes: "low, medium, high". "33% of our PRs are low-risk and are auto-approved by the bot", after which the author can merge, "which in our case reduced PR lead time by 20-40% (when compared with all PRs)".

The design is what makes the case worth copying. "The rule set for the approval bot is built based on analysis of our production incidents and the typical drivers for outages", and the rules "are highly specific to our tech stack, deployment manifests, configuration files, etc." The classes follow consequence, not size: "Typos that break configuration are assessed as high risk"; "Breaking backwards-compatibility is medium risk and requires judgement from another human"; "Documentation only changes are low risk."

The bot also changed behaviour. Zalando reports, as anecdote, that engineers began splitting work so the backwards-compatible part shipped quickly as low risk: "In the past, we observed such changes to be mixed together, increasing time to market and rollout risk." Routing rewarded small, separable changes.

What it does not show. The 20–40% compares low-risk PRs with all PRs, and low-risk PRs would be faster anyway, so part of the gain is selection. Zalando gives no escaped-defect or incident rate for the auto-approved class. And the post does not say whether the classifier uses an LLM, so this is evidence for routing, not for AI approval.

Together the two cases give the chapter's design: a precise AI reviewer for what humans still read, and incident-derived rules that take a share of changes off the human queue. Neither published the safety comparison that would make the combination L3.

Migration steps

Climb per task type: "Review: L3 for documentation-only changes; L2 for everything else" is a valid placement.

L0 → L1.

1. Allow author-side assistance: suggested fixes for reviewer comments, explanations of a diff. Nothing posts to the pull request under the AI's name.
2. Record a baseline now, before any bot arrives: time to first review, active review time, closure time, and the share of PRs merged with no review. You will need it to judge every later step.

L1 → L2.

1. Deploy a reviewer bot that comments and cannot approve. Give it no approval right in your repository settings, not just in its instructions.
2. Tune for precision before coverage. Uber's uReview filters comments by a confidence score with per-category thresholds, merges near-duplicates, suppresses whole categories developers did not value (style and readability among them), and has a second model grade the first (Uber, T3). GitHub states the same principle for Copilot code review: "Silence is better than noise" (GitHub, as of March 2026; its own figures are in the vendor box under Metrics).
3. Add a "Not useful" control on every bot comment, and write down in advance the rate at which you will switch the bot off (section 7.5).
4. Attribute bot comments so they can be counted apart from human ones.
5. After 60–90 days, compare closure time and wait time with the baseline. If closure got slower, fix precision before going further.

L2 → L3. This is the step that changes what humans review, and it needs more than a better bot.

1. Build risk tiers from your own incident history, as Zalando did. Ask which changes caused outages, and write rules for those change types. Do not start from a vendor's generic categories.
2. Define the evidence bundle a low-risk change must carry: which checks ran and passed, the AI reviewer's findings and how each was resolved, the change's risk class and why. The book's templates appendix carries a Merge-Readiness Pack template.
3. Enforce routing in the merge system (branch protection, required approvers per path), not in a policy page. Neither the authoring agent nor the reviewing AI may edit the rules or the classifier. This is the oracle principle applied to review: the routing rules are the verifier.
4. Keep a human audit sample of the routed-away class, and run the routed class for at least 90 days with its escaped-defect rate compared against human-reviewed changes of the same class. Only then call it L3.

L3 → L4, for one task class only. There is no public LLM evidence at this level, so treat it as an exception you have to argue for. The class must be narrow and reversible (documentation-only changes are the obvious first candidate). It needs a documented verification record, continuous monitoring with a disable switch, periodic audit samples, and a named owner who has signed the risk acceptance. Most teams should stop at L3, and many at L2.

Worked example: where reviewer hours go

The arithmetic below is illustrative, with no organisation behind it. It shows why the commenter and the router have different ceilings. A team receives 200 pull requests a week, and a human review takes 30 minutes.

Illustrative arithmetic. 200 PRs/week, 30 min human review each.

Scenario                                    Per-PR minutes     Hours/week
A. No AI review                             30                   100.0
B. Noisy bot: 5 comments/PR,                30 + 5 = 35          116.7
   1 min each to read and dismiss
C. Precise bot: 2 comments/PR at 1 min,     30 - 5 + 2 = 27       90.0
   saves 5 min of human review
D. As C, plus routing: 33% low-risk         134 x 27 = 3,618 min
   auto-approved, 10% of those audited      6.6 x 30 =   198 min
   at 30 min                                total      3,816 min   63.6

B vs A: +16.7%.  C vs A: -10.0%.  D vs A: -36.4%.

Row B is the industrial-study pattern: comments that cost more attention than they save. Row C's gain is bounded by how much of each review the bot can replace. Row D takes whole reviews off the queue, so its gain is bounded by the share of changes you can show are low-risk, not by comment quality. The 33% is borrowed from Zalando only as an input; your share depends on your incident history.

Guardrails

Separation of duties. The agent that wrote a change must never approve it, and neither may the person who asked for it. GitHub's Copilot cloud agent ships with this as a default: it pushes only to its own branch, its "workflows are not triggered until Copilot cloud agent's code is reviewed", and the requester cannot approve its pull request (GitHub docs, as of October 2026). The wiki's note on separation of duties for agents covers the pattern, and Security, Permissions & Blast Radius sets the full permission envelope per level.

A fresh context for the reviewer, and a limit on what it is asked. Anthropic's guidance recommends a separate reviewer session that sees only the diff and the criteria, and warns: "A reviewer prompted to find gaps will usually report some, even when the work is sound… Chasing every finding leads to over-engineering" (Claude Code best practices, vendor guidance, as of October 2026). Ask for named classes of defect, not "anything wrong"; producing what the prompt implies is the failure the wiki describes under sycophancy.

Independence is an open question. Critics argue that when one vendor's model both writes and reviews the code, the review is not an independent check. One user quoted by InfoQ said it "does not even meet minimum safety standard" (secondary, April 2026). No study measures whether model or vendor diversity catches more defects. Until one does, do not count an AI review as the independent second check that a regulated change requires.

Keep design review human. Uber says uReview "only has access to the code", not to past PRs, feature-flag configurations, database schemas or technical documentation. A bot that cannot see the design cannot review it, so changes touching architecture, data models or compatibility always route to a person. Requirements, Specs & Design covers who owns those decisions.

A pre-agreed off-switch. Google published a numeric rule for automated review feedback years before LLMs. Its Tricorder platform tracks how often developers click "Please fix" against "Not useful" on each analyzer's findings: "If the ratio for an analyzer goes above 10%, the Tricorder team disables the analyzer until the author(s) improve it", and it "has disabled an analyzer (on several occasions)". The platform's stated bar is that developers "should feel the check is pointing out an actual issue at least 90% of the time" (Sadowski et al., CACM 2018).

The book's proposal is to adopt the same mechanism for AI reviewers: a feedback control on every comment, a tracked ratio, a threshold fixed before launch, and the bot switched off, per category if possible, until fixed. Tricorder's 10% applied to static analysers; treat it as a starting point. The book's chapter on measuring success and when to roll back builds rollback triggers for every phase on this template.

Cap the batch. Zalando reports that some teams "who found large PRs to be a problem, have reached internal agreements that they will limit PR sizes to a fixed size." Size matters to an AI reviewer as well as a human one: Anthropic reports that its reviewer finds far more issues on large PRs than on small ones (vendor-reported, as of March 2026; figures in the box under Metrics). Bigger diffs produce more findings to triage. A size limit protects both reviewers.

Metrics

Every vendor reports a number that sounds like "the share of comments that were good", and no two measure the same thing. These terms keep them apart.

A vocabulary for review metrics

• Useful: a human rated the comment helpful. Opt-in, so it counts only the people who bothered to click.
• Addressed: the code changed in the same changeset in a way that responds to the comment, judged by a rule or a model.
• Led to change: a later commit touched the code the comment referred to. It does not say the change was right or needed.
• Resolved (AI-judged): a model decided at merge time that the flagged issue was fixed.
• Actionable: a property of a review, not a comment: the review posted at least one comment.
• Marked incorrect: a human explicitly flagged a finding as wrong. It is a floor on false positives, since most wrong comments are simply ignored.

None of these measures whether a defect was caught that would otherwise have shipped. That is the number you want, and nobody publishes it. Human baselines exist for only one of them: Uber's audits found 51% of human-written comments addressed in the same changeset, against more than 65% for uReview, by different methods (see Case Studies).

What to track instead

Measure your reviewer by what it does to the queue and to what ships, and pair every flow number with a stability number.

• Wait time and active time, separately, and agentic PRs separately from assisted ones. The bottleneck chapter explains why one "review time" number hides the queue.
• Closure time against your pre-bot baseline. This is the metric that went the wrong way in the industrial study.
• The "Not useful" rate per comment category, against your disable threshold.
• Escaped defects and reverts for AI-reviewed and routed changes, against human-reviewed changes of the same class. This is the L3 gate.
• The share routed to the fast lane, and the audit-sample findings from it.
• PRs merged with no review, by anyone or anything. If this rises, the queue is being relieved by skipping, not by routing.

Anti-patterns

Counting comments as value. More comments per PR, or a higher "resolved" rate, is activity. The industrial study had 73.8% resolved and slower closure. If your dashboard leads with comment counts, it is measuring the cost.

Chasing every finding. A reviewer prompted to find gaps will find some. Treating every AI comment as mandatory turns a precision problem into over-engineering.

A bot on top of an unchanged queue. Adding an AI commenter while every PR still waits for the same human approval leaves the constraint where it was, and can add to it (row B of the worked example). If the bot does not change what humans review, it has not touched the bottleneck.

Counting "AI reviewed" as "reviewed". An AI comment is not an approval, and the products say so. A process that lets a bot's pass substitute for a human approval without written routing rules, an audit sample and an escaped-defect comparison has made an undeclared L4 grant. Under the authority rule, that is the level you are at, whatever the policy says.

The same agent writes and approves. If the authoring agent can approve its own pull request, or edit the reviewer's instructions or the routing rules, there is no review.

Treating human review as an unlimited sink. Review capacity is finite, and teams that hit it tend to skip review rather than plan for it. Faros's telemetry shows more merges with no review at all (T3, vendor; see the bottleneck chapter). curl hit the limit and cut the inflow. Plan review capacity before you scale generation; if you cannot, throttle generation, as Google's migration team did.

Ranking review vendors by their own numbers. "70% resolved", "71% actionable" and "38.7% led to change" are three different measurements. Compare tools on your own pull requests instead. The wiki's code-review agents playbook and its comparison of review bots cover the tooling, and its note on review queues for agent output covers the queue. Testing & Verification covers the checks a routed change has to pass.