Code Review Agents

Do the attention arithmetic before you write any code.

A review agent is a notification system, and its budget is not tokens — it is the reviewer's willingness to keep reading. That budget is small and it is spent per comment, whether the comment was right or not.

Run the numbers on a team merging 40 pull requests a week. A bot averaging eight comments per PR at 40% precision produces about 320 comments, of which roughly 190 are wrong. Every one of those costs a developer a context switch, a read, and a judgement call. Within a month the team has learned that the fastest way to merge is to scroll past the bot — and the 130 correct comments go with it. Halve the volume to four comments at 75% precision and the same team gets 120 findings that people still read.

Review the diff, in the context of the code it touches.

The two failure shapes here are opposite and both common. Feed the agent only the unified diff and it reports on things the surrounding file already handles — a null check that exists twelve lines up, a validation the caller already performed. Feed it the whole repository and it drowns: the signal is a hundred changed lines inside a million-line haystack, and the model spends its attention summarizing architecture nobody asked about.

The shape that works is diff-anchored, context-expanded retrieval:

• Anchor on changed hunks. Every finding must cite a line the PR actually modified. A comment on untouched code is out of scope by construction, and this single rule removes a large class of noise.
• Expand to the enclosing unit. Pull the full function or class containing each hunk, not a fixed window of ±20 lines. A window that cuts a function in half manufactures false positives about missing returns and unclosed resources.
• Follow one hop of callers and definitions. For each changed symbol, fetch its definition and its direct call sites. This is where "the caller already validates this" lives, and it is the single highest-yield expansion — see repo navigation and context.
• Stop there. Two hops is almost always cost without accuracy, and it is the change most likely to blow the per-PR budget you set in agent cost control.

Give it the three inputs a human reviewer has and a diff does not.

Ask why a competent reviewer outperforms a model on the same diff and the answer is rarely reasoning ability. It is that the human knows three things the diff does not contain.

• Intent. The PR title, description, and linked issue. Without them the agent cannot tell a deliberate behavior change from a regression, and it will confidently report the former as the latter.
• History. git log and git blame on the touched lines. Code that looks wrong is often code that was fixed to look that way, and the commit message says so. A blame lookup on a suspicious line is the cheapest false-positive filter available.
• Runtime truth. Whether the tests pass, what the type checker says, what the linter already flagged. An agent that duplicates your linter is spending the attention budget on findings the reviewer has already seen in CI.

That last one is a hard rule and it is worth stating plainly. Never comment on anything a deterministic tool already reports. Formatting, unused imports, obvious type errors, and lint rules are solved problems with zero false positives; a model re-deriving them adds only the chance of being wrong. Run the deterministic tools first, put their output in the context, and instruct the agent that those findings are taken.

Make every finding survive a verification pass before it is allowed to post.

This is the step that separates a review agent people keep from one they turn off. Generation and publication are different decisions, and the gate between them should be adversarial: a second pass whose job is to refute each candidate finding using the repository, not to agree with it.

# review/gate.py — a finding earns its comment; it is not granted one
def admit(finding, repo):
    if finding.line not in repo.changed_lines:
        return "drop: not in this diff"
    if repo.linter_already_reported(finding):
        return "drop: deterministic tool owns this"
    if not finding.concrete_trigger:        # inputs + state -> wrong behavior
        return "drop: no failure scenario"

    # adversarial pass: prompted to disprove, defaults to refuted
    verdict = refute(finding, context=repo.expand(finding.line))
    return "post" if verdict.stands else "drop: refuted"

The concrete_trigger requirement does most of the work. Forcing each finding to name the inputs and state that produce wrong behavior kills the entire category of vague review comments — "this could be a race condition", "consider error handling here" — that are unfalsifiable and therefore unactionable. If the agent cannot describe how the code fails, it does not get to say that it does.

Spend a fixed comment budget, ranked worst-first.

Give the agent a hard cap — three to five comments per pull request is the range teams sustain — and make it choose. A cap is not a limitation on the agent's ability; it is what forces the ranking step that produces the quality your reviewers actually perceive.

• Rank by consequence, not by confidence. A likely-correct nitpick outranks nothing. Order by what breaks if the reader is right to ignore you: data loss, security, correctness under load, then everything else.
• One comment per root cause. The same mistake repeated in six places is one finding with six locations, not six findings. Nothing reads as machine-generated faster than the same paragraph posted six times.
• Separate blocking from advisory. Post blocking findings inline where they must be read; put the rest in a single collapsed summary comment that costs one line of scroll. Two channels with different attention prices, per progressive disclosure.
• Say nothing when there is nothing. A bot that posts "LGTM, no issues found" on every clean PR is spending attention to report the absence of information. Silence is a valid and underused output.

Instrument the one metric that predicts whether it survives.

Offline benchmarks on curated bug datasets will tell you the agent is good. Production will tell you whether anyone believes it. The number to put on a dashboard is acted-upon rate: comments that produced a code change or an explicit reply, divided by comments posted.

• Track it per repository and per finding category. It is almost never uniform — a bot with 70% overall can be at 20% on concurrency and 90% on API misuse, and the fix is to disable the category, not to tune the prompt.
• Treat a resolved-without-change thread as a labeled false positive. That is a free, continuously produced eval set arriving from your reviewers, and it is better than anything you would have written by hand; feed it back per eval-driven development.
• Watch the trend, not the level. A declining acted-upon rate means the team is starting to scroll past, and it shows up weeks before anyone files a complaint.
• Remember the agent is reading untrusted input. A pull request body or a source comment can carry instructions aimed at your reviewer — treat the diff as hostile text, per prompt injection defense.

Related: coding agent architecture for the loop this sits inside, evaluating coding agents for offline scoring, and LLM-as-judge for the verification pass in STEP 4.