Evaluating Guardrails & Detectors

Do the base-rate arithmetic first. It usually ends the conversation.

Take a detector with genuinely good numbers — 99% recall, 1% false-positive rate — and run it over a day of traffic where one request in ten thousand is actually an attack. That is not a pessimistic base rate; for most internal agents it is generous.

100,000 requests/day, base rate 1 in 10,000  →  10 attacks, 99,990 benign

  true positives   = 10     x 0.99  =  9.9
  false positives  = 99,990 x 0.01  =  999.9

  precision = 9.9 / (9.9 + 999.9) = 0.98%
  → 101 alerts per real attack, 1,000 blocked users per day

Nothing about that detector is bad. The arithmetic is doing the damage, and it is the arithmetic that a benchmark built from 500 attacks and 500 benign prompts — a 50% base rate — hides completely. A published F1 on a balanced set is a statement about the scoring function; precision at your base rate is a statement about your Monday.

The unit of evaluation is an operating point, not a detector.

Almost every guardrail is a scoring function plus a threshold, and the threshold is yours. Comparing two detectors by their default settings compares two vendors' guesses about your risk appetite.

• Report a curve, then pick a point. The comparable number is false-positive rate at a fixed recall — pick the recall the risk owner signs off on, then rank candidates by what they cost you in false alarms to get there. A single accuracy figure is not comparable between detectors and rarely comparable between releases of the same one.
• Derive the threshold from the cost ratio, not from the curve's elbow. If a missed prompt injection on a read-only summarisation agent costs a bad summary, and a false positive costs a blocked user and a support ticket, the ratio is not obviously in favour of catching everything. On an agent holding write credentials it inverts hard. Same detector, two thresholds, and the decision is a product one.
• Different thresholds per surface. One threshold across a whole platform is a sign nobody did this exercise. Tier by what the agent can do with the input, which is the same axis as autonomy levels.
• Score the classes you will act on differently. A detector that fires "unsafe" is nearly useless operationally; one that fires "unsafe: instruction-in-retrieved-content" routes. Evaluate per class, because per-class recall varies enormously and the aggregate hides it — the same argument as failure taxonomies.

Your base rate is not the benchmark's, and you have to measure it.

You cannot compute precision without a prevalence estimate, and there is exactly one honest way to get one: label a uniform random sample of production traffic. Not the flagged sample — the flagged sample is where the detector already looked.

• Sample uniformly, label blind, and keep the sample. A few hundred requests drawn at random, labelled by someone who cannot see the detector's score, gives you prevalence and an unbiased false-negative estimate at the same time. Everything else you measure is conditioned on the detector's own opinion.
• Re-estimate on a schedule. Base rate moves when you launch in a new market, expose a new surface publicly, or get named in a write-up. A threshold tuned at one prevalence is mistuned at another, and nothing in the system will tell you.
• Expect the confidence interval to be embarrassing. If prevalence is genuinely 1 in 10,000, a 500-request sample will contain zero attacks and your interval is "somewhere under 6 in 1,000". That is a real result: it tells you the false-positive side dominates the design and that you should stop arguing about recall.
• Count what a miss actually costs, in incidents. The number that justifies a tighter threshold is not recall on a benchmark; it is the count of production incidents whose first-wrong-step was an input this detector should have caught. If that count is zero over two quarters, the detector is not the control doing the work.

A frozen benchmark measures a distribution your attacker is allowed to move.

Guardrail evaluation has a property ordinary quality evaluation does not: the input distribution is chosen by someone who wants your number to be wrong. Public injection corpora are the ones every vendor tunes against and every attacker has read, which makes a high score on them the weakest possible evidence.

• Keep two sets and never mix them. A frozen regression set, versioned and unchanged, exists to detect that an upgrade made things worse; it is meaningless as an absolute measure. A rotating adversarial set, refreshed by a red team each cycle against the current threshold, is the only thing that estimates present-day recall.
• Grade the adversary's effort, not just the outcome. "Three attempts to bypass" and "two hundred attempts to bypass" are different products at the same recall. Record attempts-to-first-bypass; it is the metric that degrades visibly as a detector ages.
• Hold out the payloads you buy. If a vendor's benchmark ships in their training set, their score on it is a memorisation measurement — the guardrail version of benchmark contamination. Reserve a private set you never send them.
• Include the boring negatives. Half the false positives in production come from legitimate text that looks adversarial: security documentation, prompt-engineering discussions, bug reports quoting an error, a customer pasting a log. Get those into the benign half or your false-positive rate is fiction.

The detector has a latency budget and a failure mode, and both are usually unmeasured.

A guardrail sits on the critical path of every request, twice if you scan input and output. Its quality numbers are only half the evaluation; the other half is what it does to the system it protects.

• Measure added p95, not added mean. A classifier averaging 80 ms with a 900 ms tail turns a snappy agent into an occasionally broken one, and on a multi-step run you pay the tail several times. See measuring agent latency.
• Decide fail-open or fail-closed explicitly, then test it. Most stacks fail open by accident — the detector times out, the exception is swallowed, the request proceeds unscreened, and no metric records that screening did not happen. Emit a distinct counter for "not evaluated" and alert on it; an outage of the guardrail is a security event, not a latency blip.
• Price it per protected request. A model-based detector on input and output can add a meaningful fraction of the bill; a cheap deterministic pre-filter that removes 90% of traffic before the expensive check is usually the better architecture and it is invisible in any accuracy comparison. Related: agent cost control.
• Track the human cost of a block. Blocked-then-appealed requests, support tickets, and the retry rate of a user who was told no. This is the number that decides whether the control survives its first quarter.

Ship it in shadow, then govern it as a control with a stated job.

The rollout sequence is the part teams skip, and it is where the threshold actually gets chosen — on real traffic instead of on a slide.

• Shadow mode first, for at least a full weekly cycle. Score everything, block nothing, and read the top hundred scores by hand. You will find the false-positive clusters this way and no other, and you will get your prevalence estimate for free.
• Enforce on the narrow surface first. Turn it on where the blast radius is largest and the volume is smallest — the agent with write credentials, not the public chat box.
• Put the operating point in version control with an owner and a review date. A threshold is a configuration change with a security effect; it belongs in the same review path as a permission change, and it should be re-derived when the base-rate estimate is refreshed.
• State the job honestly in the risk register. A content detector reduces the rate of compromise; it does not bound the duration, because its miss is silent. The control that bounds duration is behavioural — see detecting agent compromise. Quoting a recall figure as though it were coverage is how a stack ends up with one layer and a good feeling about it.

Related: guardrails in production for the layered design this measures, quality regression detection for the frozen-set half of the discipline, and calibration for what a score has to mean before a threshold on it means anything.