Review Queues for Agent Output

Do the arithmetic before you design anything.

An agent producing 400 reviewable items a day into a team of three reviewers who each sustain — realistically, across a working day with meetings and fatigue — about 40 careful decisions an hour, gives you roughly 960 decisions of capacity against 400 items. Comfortable. Double the agent's throughput twice and you are underwater, and the failure does not look like a design problem. It looks like a backlog, then like reviewers going faster, then like approvals that were never really reviews.

Three numbers belong on a dashboard from day one, and only the first is usually there:

• Queue depth and its slope. Depth alone is ambiguous. A queue growing steadily is a capacity problem you can forecast a week ahead; a spiky one is a batching problem you can fix in the dispatcher.
• Seconds per decision, by item type. The lever you actually control through design. Track the distribution — a median of 20 seconds with a p90 of four minutes means one item type is doing all the damage, and that type is where the design work goes.
• Time from agent output to decision. What the end user experiences. In many products this is the real latency of the feature, and the model's response time is a rounding error inside it.

First-in-first-out is the wrong policy, and it is everyone's default.

Arrival order is a fairness rule borrowed from ticketing systems, where the person waiting is the one you are being fair to. Here nobody is waiting in line; the thing you are rationing is attention, and the right policy is to spend it where it is most likely to change an outcome. An item the agent got confidently right and an item where it was one fact away from a costly error are worth wildly different amounts of a reviewer's minute, and FIFO prices them identically.

# queue/priority.py — recompute on enqueue and on model version change
def priority(item):
    # expected value of review = how likely we are to flip it,
    # times what flipping it is worth, minus what it costs to look
    p_flip = 1.0 - item.confidence          # calibrated, not raw logprob
    stakes = item.blast_radius * item.reversibility_cost
    effort = item.expected_review_seconds
    return (p_flip * stakes) / max(effort, 1)

The three terms each earn their place. Probability of flipping needs a calibrated confidence, not a raw model score — if your confidences are not calibrated, start with a proxy you trust, such as whether the agent had to retry or whether a tool returned an error mid-run. Stakes must include reversibility: an action you can undo cheaply deserves less review than an identical action that sends an email. Effort in the denominator is what stops the queue filling with high-stakes items that each take fifteen minutes while forty cheap flips go unmade.

Two guards keep this honest. Age must eventually dominate, or low-priority items starve forever and someone finds a three-week-old approval. And a small random sample must bypass the ordering entirely — otherwise you only ever observe the items the model was unsure about, and you learn nothing about the confident ones, which is precisely where a silent regression lives.

Design for deciding, not for reading the agent's homework.

The instinct is to give the reviewer everything: the full trajectory, every tool call, the reasoning. It feels transparent and it is a trap. Reading a long trajectory takes minutes and does not reliably improve the decision, because the reviewer ends up checking whether the process looked sensible rather than whether the answer is right — and a fluent wrong answer has a fluent wrong process attached.

Build the screen around the decision instead:

• The claim, stated flatly, at the top. What the agent proposes to do or has concluded, in one line, in the reviewer's vocabulary rather than the system's.
• The two or three facts that would change it. The retrieved passage the conclusion rests on, the field it came from, the value it contradicts. This is the reviewer's actual work, and surfacing it is the difference between a twenty-second decision and a four-minute one.
• Provenance one click away, not on screen. The full trajectory must exist and must be reachable, because the tenth item of the day is the one that needs it. It should not be the default view. Related: progressive disclosure.
• Disagreement as cheap as agreement. If approving is one key and rejecting opens a modal with a mandatory free-text box, you have built an approval machine and your data will show a beautiful approval rate.

A 98% approval rate is a broken control, not a good model.

Approval rate is the queue's vital sign, and it is nearly always read backwards. High approval feels like validation of the agent; past a threshold it means the reviewer has stopped generating independent information. Whether that is because the agent genuinely became reliable or because a human being asked to press approve four hundred times a day has adapted the way humans do, the queue has stopped being a control — and from the outside those two causes look identical.

Distinguish them with instrumentation, not intuition:

• Seed known-bad items. A small, tracked rate of deliberately incorrect items with a recorded ground truth. Catch rate on seeds is the only direct measure of whether review is functioning, and it is the number to put next to approval rate so neither is read alone.
• Watch dwell time against approval rate. Approval rising while seconds-per-decision falls is the signature of rubber-stamping. Approval rising while dwell time holds is the agent actually improving.
• Compare reviewers. Persistent divergence between two people on the same item mix is either an ambiguous policy or a training gap, and both are fixable — but only if the queue records who decided what.

When approval is legitimately high, the response is to review less, not to keep the ritual. Move that item type to sampled review, keep the seeds running, and spend the freed capacity on the types where reviewers still change outcomes. That is the ratchet in progressive autonomy, and the queue's own metrics are what earn each click of it.

It is a professional tool used for hours, so build it like one.

Review queues get designed like consumer surfaces — generous whitespace, one card at a time, a mouse-driven approve button — and then handed to someone who will sit in them for six hours. The people who do this work end up wanting what people who do high-volume decision work always want.

• Keyboard-first, with no mouse in the common path. Approve, reject, skip, next. A reviewer who never leaves the home row is roughly twice as fast as one who reaches for a mouse each item.
• Prefetch and no full-page transitions. A two-second page transition between items is over thirteen minutes lost across a day of 400 decisions, and worse, it breaks the rhythm that makes the work sustainable.
• Batch the identical. When forty items share a cause — one bad retrieval source, one broken tool — let one decision resolve all forty, with the group visible before it is applied. This is the single largest throughput win available and it requires clustering by cause at enqueue time.
• Undo, not confirmation. A confirmation dialog taxes every correct decision to prevent a rare wrong one. A ten-second undo taxes none of them. See undo and reversibility.
• Save position, always. Reviewers are interrupted. Losing a place is a small thing that quietly makes the queue something people avoid.

A rejection that only stops one item is a wasted signal.

The queue is the highest-quality labelled data your system will ever produce: a domain expert judging real production output at the moment of use. Most implementations write a boolean and throw the rest away, which means the same failure arrives tomorrow and is rejected again by hand.

• Capture a reason from a short, closed list. Five to eight causes drawn from your actual failure taxonomy, one key each, with free text optional and never required. Closed lists produce data you can group; mandatory free text produces empty strings and slower decisions.
• Route reasons to different owners. "Wrong source retrieved" is a retrieval bug, "policy says otherwise" is a prompt or policy change, "the tool returned stale data" is neither. A single rejection bucket hides all three.
• Promote rejections into the eval set. A rejected item with its correction is a test case, and this is the cheapest path to a regression suite that reflects production rather than your imagination — the mechanism in production feedback signals.
• Close the loop visibly. Tell reviewers what changed because of their rejections. The queue's throughput depends on people who believe the work matters, and nothing degrades review quality faster than the sense that the decisions vanish.

Related: approval & confirmation UX for the in-line case where the user is waiting, async agent UX for the surrounding workflow, human-in-the-loop for what the control is meant to guarantee, and annotation & labeling ops for running the reviewer pool as a programme.