Measuring Calibration in Production

Step 1Choose the outcome before you choose the metric.

Calibration is a comparison between a number the system emitted and something that later turned out to be true. In an offline evaluation the second half is free — you have labels. In production you have to nominate an observable event and defend it, and that nomination determines everything downstream. Ranked by how much you can trust them:

• A deterministic check. The SQL ran, the schema validated, the test passed, the invoice total reconciled. Unbiased, cheap, available on every item — and only exists for a minority of tasks.
• A human edit or revert. The reviewer changed the draft, reverted the commit, overrode the classification. Available broadly, but censored: it only observes items a human looked at, and the whole point of a confidence badge is to stop humans looking at some of them. Measure calibration from reverts alone and you measure it exactly where the badge is not operating.
• A delayed business event. The customer reopened the ticket, the payment was disputed, the appointment was a no-show. Closest to what you care about, and the lag is weeks, so it cannot close your deployment loop.

The practical answer is a deliberate sampling arm: route a fixed, small fraction — 2–5% is usually enough — to full human review regardless of what the confidence said, and compute calibration on that arm. It costs real money and it is the only label source that is not missing-not-at-random. Everything else is a supplement. The arm is also what makes online evaluation comparable to your offline numbers rather than a different quantity with the same name.

Step 2ECE moves when you change a free parameter.

Expected Calibration Error is the default headline: bucket items by confidence, take the gap between mean confidence and observed accuracy in each bucket, weight by bucket size. It is easy to compute and it has a property that makes it dangerous as a tracked metric — the bin count is yours to choose, and the answer depends on it.

Here is a thousand-item sample, grouped by the confidence the surface displayed. The values cluster on round numbers because verbalized model confidence does; it is reported as 90% and 95% and “certain”, not as 0.873.

# 1,000 production items, grouped by displayed confidence
#   conf    n     accuracy    gap (conf - acc)
    0.70    80      0.65         +0.05
    0.80   140      0.72         +0.08
    0.90   300      0.78         +0.12
    0.95   280      0.99         -0.04
    1.00   200      0.74         +0.26

# Same data, two defensible bin counts:
  ECE, 10 equal-width bins .......... 0.092
  ECE, 20 equal-width bins .......... 0.114     (+24.3%)

# Why: at 10 bins, 0.95 and 1.00 share the top bin, where
# 280 under-confident items cancel 200 over-confident ones:
  merged bin: n=480, mean conf 0.971, accuracy 0.886

The mechanism is worth stating plainly, because it is not noise and it is not sampling error. ECE takes an absolute value per bin, so a bin that merges an over-confident group with an under-confident one reports their cancellation as calibration. Choose ten bins and the top of your scale looks 24% better than it is. Nobody has to act in bad faith for this to happen; ten is the conventional default.

Two consequences. First, never compare an ECE figure to one computed elsewhere unless both state the binning — a vendor's ECE and yours are not the same measurement. Second, if you track ECE over time, freeze the bin count in the code and treat changing it as a metric migration, the same discipline as a change to an eval set.

Step 3Track Brier's reliability term, because squares do not cancel.

The Brier score — the mean squared difference between the forecast probability and the binary outcome — has no bin parameter, and it decomposes into three terms that each tell you to fix a different thing. On the same thousand items:

# Brier = reliability - resolution + uncertainty
  reliability ....... 0.0194    miscalibration: how far conf is from acc
  resolution ........ 0.0135    discrimination: how far bands are from base rate
  uncertainty ....... 0.1527    irreducible: base_rate * (1 - base_rate)
  ------------------------------
  Brier ............. 0.1585    base rate (overall accuracy) 0.812

# Where the miscalibration lives: contribution to reliability
  conf 1.00 band (200 items, 20% of volume) ..... 70% of reliability

Each term answers a separate question. Reliability is calibration proper — it falls when your displayed numbers match observed accuracy, and it is the term a recalibration fixes. Resolution is discrimination — it rises when your confidence bands actually separate easy items from hard ones, and it is the term a better model or a better signal fixes. Uncertainty is a property of the task's base rate and you cannot move it at all; it is why a raw Brier score is incomparable across task types.

The reason to prefer reliability over ECE as the tracked number is in the arithmetic above: it squares the gap per forecast value, so opposite-signed errors add rather than cancel. That is also what surfaces the finding that matters here — 70% of the total miscalibration sits in the 20% of items the system called certain. No ECE at any bin count tells you that; it is a per-band quantity and you have to report it per band.

Step 4Report the signed gap per band, and read it as a review budget.

One aggregate number hides direction, and direction decides the fix: over-confidence needs friction added, under-confidence needs friction removed. Report the signed mean gap (here, +0.092 overall) and report it per band and per task class, because the aggregate is a weighted average of opposite problems.

The band table also tells you where reviewer attention is being wasted, which is the version of this report an operations lead will act on. Take the two top bands from STEP 2:

• The 0.95 band: 280 items, 99% accurate — 2.8 errors. Reviewing all of it costs 100 reviews per defect found.
• The 1.00 band: 200 items, 74% accurate — 52 errors. Reviewing all of it costs 3.8 reviews per defect found, 26 times more efficient.

Almost every product does the opposite, because the band labelled “certain” is the band whose UI suppresses confirmation. The inversion is not a reviewer-discipline problem; it is manufactured by the display. The trust-side argument is in designing for trust, and the throughput arithmetic — why misallocated review exhausts the budget before reaching what needed it — is in oversight capacity.

Segment before you believe any of it. A single calibration curve over all traffic can look excellent while every segment inside it is badly calibrated in opposite directions; this is the ordinary aggregation trap, and the segments that matter are task class, input language, tenant, and model version. The failure-taxonomy discipline in failure taxonomy and triage gives you the task classes for free if you already have one.

Step 5Recalibrate the display, not the model.

You do not need to retrain anything to fix reliability. You need a monotone map from the raw signal to a displayed number, fitted on the logged pairs from your sampling arm — isotonic regression when you have a few thousand labelled items, Platt scaling when you have hundreds. Fit it on held-out data, version it alongside the model, and re-fit when either the model or the prompt changes, because a recalibration is only valid for the system it was fitted to. Treat an unpinned model default as invalidating it: see unpinned vendor defaults.

Then choose what the surface actually shows, which is a separate decision from what you computed. Three states beat a percentage almost always, and the useful third state is the one most products omit:

• Checked and it held — a verifier ran and passed. Backed by substrate, not by the model's opinion of itself.
• Checked and it did not hold — the verifier ran and failed, or the citation does not support the claim.
• Not checked — no verifier exists for this item, or the verifier errored. This is the state that gets silently folded into one of the other two, and folding it is how a badge starts lying.

TruffleHog's result model is the clean implementation of exactly this distinction — verified, unverified, unknown, where unknown means verification itself failed rather than that the finding was dismissed. It is a security tool rather than an agent surface, and it is still the best available reference for an honest confidence display, because that team was forced to be explicit about what it had and had not checked.

Step 6What a calibration monitor should page on.

Calibration drifts without anything breaking: the input mix shifts, a tenant onboards with harder documents, a provider changes a default. A monitor is therefore not optional, and it should alarm on four things — three of which are not calibration numbers at all.

• Signed gap in the top band crossing a threshold you set from the review budget, not from a round number. This is the one that corresponds to users being misled.
• Label coverage falling. If the sampling arm stops producing labels — reviewers behind, queue rerouted, a flag turned off — every calibration figure downstream silently freezes at its last good value. Alarm on the absence of labels, the same way monitor integrity alarms on the absence of a synthetic finding.
• Distribution shift in the displayed confidence itself. If the share of items in the top band jumps, something changed upstream, and your fitted map is now being applied outside the range it was fitted on.
• Recalibration age. A map fitted against a model version that is no longer serving is a stale control. Page on the mismatch, not on the number it produces.

The trap to avoid is automatic re-fitting on a rolling window. If labels are missing-not-at-random — and under any review-suppression policy they are — a nightly re-fit will converge toward the confidence distribution of the items humans still happen to look at, which drifts away from the population the badge is shown to. Re-fit on the sampling arm only, deliberately, with the change reviewed like a model change.