Uncertainty & calibration.
Every escalation rule, routing decision and "ask a human" gate in your system rests on a confidence number the model was never trained to produce honestly — and the alignment step that made it pleasant to talk to is the step that broke it. Three different signals get called confidence, only one of them is worth much, and none of them arrives calibrated for your data. The good news is that fixing this is an afternoon of work with a labelled set, not a research project — and the thing you build is not a number to display but a threshold that decides when the agent stops.
Three different signals share one word.
"How confident is the model?" has at least three mechanical answers, and they measure different things. Conflating them is the origin of most confidence bugs, because each one is plausible-looking and only one of them tracks whether the answer is right.
- Token probabilities. The model's own probability for each token it emitted. Genuinely informative — but about the next token given the prefix, not about the truth of the claim. Average the logprobs over a paragraph and you have largely measured fluency: a smoothly-written fabrication scores high, and a correct answer phrased awkwardly scores low. They are also frequently unavailable, since several providers do not expose logprobs at all on reasoning models, and in an agent the decision you care about is often a tool call rather than a span of prose.
- Verbalised confidence. Asking the model to say how sure it is. This is a writing style learned from text, not a measurement: outputs cluster on round numbers, skew high, and move when you rephrase the question. It is nearly free, so it is worth collecting — but treat it as a weak feature to be calibrated, never as a probability.
- Agreement across samples. Run the same input k times at non-zero temperature and measure how much the answers agree. This is the most predictive of the three by a wide margin, because disagreement is direct evidence that the question sits where the model's distribution is genuinely spread out. It costs k times as much, which is exactly why you spend it only on decisions worth that. It is the same machinery as self-consistency, read for a different purpose.
A fourth signal is often better than all of them and gets overlooked: whether the claim is grounded. "Did the retrieved documents actually support this sentence?" is a checkable question with a cheap answer, and it beats any introspective confidence estimate on exactly the failure everyone is worried about. See hallucination and grounding.
Calibration is a measurable property, and post-training destroys it.
A system is calibrated when its stated confidence matches its observed accuracy: of all the times it says 70%, it is right about 70% of the time. That is checkable with nothing more than a labelled set — bucket the predictions by stated confidence, plot accuracy against confidence, and look at the gap. The summary statistic is expected calibration error, the average distance between the two across buckets, but the plot is more useful than the number because it shows you where the model lies.
- Base models are often decently calibrated, and aligned models are not. OpenAI reported this directly for GPT-4: the pre-trained base model's confidence tracked its accuracy closely on multiple-choice knowledge questions, and the post-trained, RLHF'd version was visibly worse. That result has held up as a general pattern rather than a quirk of one release.
- The mechanism is not mysterious. Preference optimisation rewards answers that human raters like, and raters like decisive, fluent, helpful-sounding text. Hedging reads as evasive and loses. You asked for a model that sounds sure of itself and you got one. This is one of the standing costs of post-training.
- Therefore you cannot inherit calibration from the vendor. Whatever calibration a model has is calibration on its evaluation distribution, not on your tickets, your documents or your tool-call schema. The mapping from raw score to real probability is yours to fit.
- Fitting it is small. Collect a few hundred outcomes with ground-truth labels, then fit a one-parameter temperature scaling or an isotonic regression from raw signal to probability. This is a dozen lines and it is the difference between a number you can put a threshold on and a number you cannot.
The useful output is an abstention, not a percentage.
Teams spend weeks improving a confidence estimate and then do nothing with it, which is the wrong order. The reason to want calibrated confidence is to build a gate: below some threshold the agent stops and hands the task to a human, a stronger model, or a deterministic fallback. That framing — selective prediction — turns an abstract statistical property into a decision you can price.
- Draw the coverage–risk curve, then choose a point on it. Sweep the threshold; at each one record coverage (the share of inputs you answer) and risk (the error rate among those answered). You now have the actual trade-off in front of you rather than a vibe about it.
- State the result as a business sentence. "At the threshold that holds error under 2%, we answer 61% of tickets automatically and route 39%" is a sentence an operations lead can act on. "Our expected calibration error is 0.04" is not, and it is the sentence that makes people ignore this work.
- The abstention needs somewhere to go. A gate that fires into a queue nobody reads is worse than no gate, because it converts a visible error into an invisible delay. Decide the destination first — see human in the loop — and only then tune the threshold.
- Prefer three outcomes to two. Answer, escalate, and decline with a reason are different, and the third is often the honest one: "I could not find a policy clause covering this" is more useful to everyone than a hedged answer or a silent handoff. Make it a field in your structured output so downstream code can branch on it.
- Thresholds are per-task, not global. The confidence at which you let an agent answer a billing question and the confidence at which you let it issue a refund are different numbers, because the cost of being wrong is different. One global threshold means you have set it for your riskiest action and are paying in coverage everywhere else.
What this looks like in a running agent.
In production the confidence signal is rarely one thing. The cheap version runs everywhere and the expensive version runs where the stakes justify it, and both feed the same calibrated threshold.
- Escalate on the cheap signal, confirm with the expensive one. Use a low-cost feature — grounding check, verbalised confidence, a small classifier trained on your own traces — to select the small fraction of runs worth re-sampling, then use sampling agreement on those. This gets most of the accuracy of k samples at a fraction of k times the cost, and it is the same shape as model routing.
- Calibrate your judges too. If an LLM scores your agent's output, that judge has exactly the same overconfidence problem and its scores need the same treatment before you gate on them — judge calibration is not optional if the judge decides anything.
- Recalibrate on every model change. A calibration fitted on one model version does not transfer to the next one, and a silent provider-side update will move it without telling you. Add the recalibration step to the migration checklist in model deprecation and migration, and re-run it whenever your prompt changes materially.
- Resist displaying the number. A confidence percentage on the screen invites the user to do the arithmetic you should have done in code, and it reads as precision the estimate does not have. Use it to route; show the user the evidence instead. The UX case is made in designing for trust.
Do the one-afternoon version before you do anything sophisticated: take 200 completed runs where you know the real outcome, score each with whatever signal you already have, sort by that score, and plot the error rate as you extend down the list. You will find the threshold that holds error where you need it, and you will learn the coverage it costs. That single plot is worth more than any improvement to the confidence estimate itself, because it is the artifact that turns "the model seems unsure" into a rule your system can execute.
Related: evals 101 for where the labelled set comes from, agent evaluation for the trajectory-level version of the same question, and agentic risks for why a confidently wrong agent is the expensive one.