Detecting Quality Regressions

Why the obvious monitor does not exist.

Offline evals answer "is the new version better on our suite" before release. That is a different question from "is the thing running right now still working", and the second one is the one that pages you. Three structural facts stand between you and a direct accuracy alarm:

• There is no ground truth in production. The user asked something nobody has a labelled answer for. Anything you compute is an estimate produced by another model, with its own drift.
• Judging is a second production system. Running an LLM judge on every run roughly doubles your inference footprint and adds a component that can itself regress silently when the judge model is updated. Nobody runs it at 100%, so everybody samples, and sampling costs you time.
• Statistical power is unkind at high baselines. The better your agent is, the more traffic it takes to prove it got worse:

# Detect a drop from 90% to 85% success, alpha 0.05, power 0.80
n_per_arm = 700                # ~1,400 scored runs in total

# At 2,000 runs/day:
#   judging 100% of traffic  ->  detectable inside a day, at 2x inference
#   judging 5% of traffic    ->  detectable in about two weeks
#
# Over that same window, the tool-error rate leaves its
# 14-day band within an hour, for free, with no labels.

None of this argues against judged evaluation — it is the confirmation step, and step 4 puts it back. It argues that judged evaluation is the wrong detector. See eval variance and statistical power for why these numbers are worse than intuition suggests, and online vs offline evals for the division of labour.

The shape of the run moves first.

An agent that has gotten worse almost always works harder before it answers worse. Retrieval degrades, so it searches three more times. A tool schema changed, so it retries and reformulates. The model update made it more cautious, so it asks a clarifying question it used to skip. All of that is visible in the trace before a single output is judged, and all of it is free because you are already emitting spans.

• Steps per run, as a distribution. The mean barely moves while the tail doubles — and the tail is where the failures are. Watch p50, p90 and the fraction of runs that hit the step cap; that last one is the single most informative number on this list.
• Tool-call error rate, per tool. Aggregated, it hides everything. Per tool, it names the broken integration in the alert itself.
• Retry and reformulation rate. The same tool called twice with near-identical arguments is the agent telling you the result was unusable.
• Termination-reason mix. Completed, gave up, hit the step cap, hit the token cap, errored, escalated to a human. This mix is remarkably stable in a healthy system and it moves days before anyone complains.
• Context length at the final turn. A creeping upward drift means retrieval is returning more and cutting less, which is both a cost signal and a quality one — long contexts are where instruction-following decays.
• Refusal and clarification rate. The most sensitive detector of a silent model update there is, and it needs no labels at all.

Alert on bands rather than thresholds: compute a rolling fourteen-day distribution per statistic and page when today sits outside it, segmented by tenant, entry point and agent version. A fleet-wide average is exactly the aggregation that lets one broken segment hide.

Add invariants, which are cheap and unambiguous.

Trajectory statistics are sensitive but soft — they tell you something changed, not that something is wrong. Invariants are the opposite: rare, but when one fires it is a defect with no interpretation required. Every agent has a handful, and they are the cheapest quality signal you will ever ship.

• Structured-output validation failures. A response that does not parse against its schema is a bug, full stop. If you use constrained decoding this should be flat zero, which makes any non-zero value an incident.
• Loop detection. The same tool, the same arguments, three times in one run. Never legitimate, trivially detectable, and the leading indicator of a cost incident as much as a quality one.
• Citation grounding. For any answer that cites retrieved material, assert that the cited chunk was actually in the context. A drift here means the model is inventing provenance, which is worse than being wrong.
• Policy-gate hit rate. If your agent has a gate that blocks certain actions, its firing rate is a behavioural fingerprint. A gate that suddenly stops firing usually means the agent stopped attempting the action, not that it got safer.
• Tool-argument distribution. A parameter that has always been one of four enum values and is now sometimes free text tells you a prompt or schema change landed somewhere upstream.

Judging is the confirmation, and you stratify it by the anomaly.

Once a band is breached you need to know whether output quality actually moved, and that does require a judge. The mistake is running the judge as a uniform background sample, which is both the most expensive way to buy statistical power and the least likely to contain the failure.

• Sample where the anomaly is. If the step-cap rate doubled, judge the runs that hit the cap, not a random slice of everything. Conditioning on the anomaly raises the density of the failure by an order of magnitude, which is what makes a small sample decisive.
• Judge pairwise against the previous version, not absolutely. Absolute scores drift with the judge; a preference between two answers to the same input is far more stable and needs fewer samples for the same confidence.
• Pin and version the judge. A judge on a floating model alias will silently redefine your metric during an incident, which is the worst possible moment. Treat the judge like any other production dependency — see model deprecation and migration.
• Score the trajectory as well as the answer. A run that reached the right answer through three failed tool calls is a regression that outcome-only judging scores as a pass. Outcome vs trajectory evaluation is the full argument.
• Keep a human in the sample. A small, standing human review of judged runs is what tells you the judge itself has drifted. It is always the first thing cut and always the thing you needed.

Run a canary suite on a schedule, because your inputs move too.

Production statistics confound two things: your system changing, and your traffic changing. A Monday spike in step count might be a regression, or it might be a new customer whose documents are longer. The only way to separate them is a fixed set of inputs run on a schedule against production, where anything that moves is unambiguously yours.

• Twenty to fifty representative tasks, run hourly or daily against the live stack. Real tools, real indexes, real model endpoints. Not the offline suite with mocks — the point is to catch the dependency that changed underneath you.
• Expect nondeterminism and design for it. Identical requests do not return identical text, so a canary that asserts on exact output will be red permanently and then ignored permanently. Assert on the invariants of step 3 and on band membership for the statistics of step 2. Reproducibility and nondeterminism explains why this is structural rather than a configuration mistake.
• This is the detector for the silent third-party change. A provider ships a minor model update, an MCP server changes a tool description, an index finishes a re-embed, a vendor's rate limits tighten. None of those appear in your deploy log, and the canary is the only instrument that sees them the same day.
• Include an adversarial slice. A handful of injection attempts and known-bad inputs, permanently in the suite, so that safety behaviour is monitored on the same cadence as quality rather than at review time.

Wire the alert to an action, or do not send it.

A quality alert that arrives with no attached decision trains people to close it. The value of everything above is realised in the twenty minutes after the page, and that requires two things you must build before the incident: the ability to attribute the change, and the ability to undo it.

• Every alert names the version. Prompt version, tool-schema version, model ID, index version, retrieval config. If a regression cannot be attributed to a change, the responder's only move is to guess. Rollout and versioning is the prerequisite for this entire page.
• The first response is to pin, not to debug. Roll back to the last known-good configuration, confirm the statistic returns to its band, and diagnose afterwards. A feature flag per behaviour makes this a decision rather than a deploy.
• Know which alerts justify the kill switch. A grounding-invariant breach on a customer-facing agent and a five-point quality dip are not the same class of event; write the mapping down before you need it, and keep the switch scoped so one segment can be stopped without stopping the fleet.
• Close the loop into the offline suite. Every confirmed regression becomes a case in the eval set, or you will detect the same failure again next quarter. This is the mechanism by which a monitoring practice compounds instead of merely repeating.

Related: tracing and observability for the spans everything here reads from, agent observability for the underlying model, and incident response for agents for what happens after the page.