Designing for Trust & Calibration

7 min read

H1
Playbook · Agent UX & Human Interaction

Designing for trust means engineering for the right amount of it, not the most.

The expensive agent failures in production are not the distrusted ones — users simply stop using those. They are the over-trusted ones: confidently wrong, believed, and shipped onward. Trust is calibrated by exactly three disclosures — what you checked, what you did not check, and what your checks skip — and the best worked examples of all three in 2026 are not chat interfaces but security gates, because those teams were forced to publish their own coverage gaps.

Step 1The target is calibration, not maximization.

Two failure modes bracket every agent. Under-trust: the user re-checks everything, the speed advantage evaporates, the product is abandoned. Over-trust, sometimes called automation complacency: the user stops inspecting, so errors pass straight through to consequences. The design goal — the term of art carried from the human-automation literature into agent UX — is trust calibration: user-perceived reliability should track measured reliability, per task class, because an agent that is excellent at summarizing and mediocre at arithmetic has two true reliabilities and deserves two levels of trust.

The word doing the work is measured. Calibration is a comparison, so a product that has never computed the second half of the comparison cannot claim to be calibrated — it can only claim to feel careful. That is the difference between this page and a design-principles slide, and the rest of it is about the three things you have to disclose for the comparison to be possible at all.

“Users love it” is not a success metric for an agent. An agent users love and never verify is a latent incident. Measure the gap between trust and reliability, not trust alone.

Step 2Over-trust is the default, because fluency is the only signal users have.

Error cost is not symmetric. A hedged wrong answer — “I'm unsure, possibly X, verify” — costs a re-check. A confident wrong answer costs the full downstream consequence plus the erosion of every future calibration the user makes. And the asymmetry is self-inflicted, because language models produce equally fluent prose whether right or wrong, so fluency cannot be a reliability signal — yet by default it is the only one on the screen.

The shape of the damage is arithmetic, not vibes. Take a thousand real outputs bucketed by the confidence the product displayed, with the accuracy each bucket actually achieved:

  • The band shown as 95% confident: 280 items, 99% accurate — 2.8 errors. Reviewing all of it costs 100 reviews per defect found.
  • The band shown as certain: 200 items, 74% accurate — 52 errors. Reviewing all of it costs 3.8 reviews per defect — 26 times more efficient.

Seventy per cent of the total miscalibration in that sample sits in the twenty per cent of items the system called certain. And the “certain” band is precisely the one where every sensible-looking UI suppresses the confirmation step. The inversion is not a reviewer-discipline problem you can train away; it is manufactured by the display, which is why the fix belongs to design rather than to process. The throughput consequence — misallocated review exhausts the budget before reaching the step that needed it — is oversight capacity.

Step 3Three disclosures, and the third is the one everybody drops.

A calibrated surface tells the user three separate things. Most products tell them one and a half.

  • What was checked, and what the check was. “This figure was retrieved from the Q3 filing” is a disclosure. A bare 62% is not: it names no substrate, so the user cannot tell whether it reflects a passing verifier or the model's opinion of itself, and those have wildly different track records.
  • What was not checked. The honest third state. Most surfaces collapse “we checked and it failed” into the same bucket as “no checker exists for this”, and the collapse always resolves in the optimistic direction, because an unflagged item reads as fine.
  • What the checking itself skips. Every verifier has a coverage boundary — an input it cannot parse, a size above which it gives up, a class of claim it does not evaluate. If that boundary is undisclosed, the user's trust is calibrated against a check they believe is universal, which is worse than no check, because they stop sampling.

The third disclosure is the one with no established UI pattern, and it is where the real exposure lives. A verifier that silently declines on 4% of inputs and is believed to cover 100% produces a confident, unflagged stream of exactly the items nobody validated — the same structure as a concealed failure, except that the concealment is a product decision rather than a model behaviour.

Step 4The best implementations shipped in security tooling, not in chat.

Three products get one of the three disclosures visibly right, and all three are secret-scanning tools — a field where regulators, auditors and incident reviews forced teams to state what their checks do not do. As of November 2026:

  • Three states, honestly separated. TruffleHog returns verified (the credential was tested against the issuing API and is live), unverified (detected, not confirmed) and unknown (verification itself failed — a network or API error). unknown is the third disclosure as a first-class value, not an omission. That is the pattern to copy into any agent surface that runs a checker which can itself fail.
  • A bypass that is allowed, and recorded. GitHub's push protection blocks a commit containing a detected secret and lets a developer proceed with a stated reason, which is logged. Friction on the irreversible step, with the override made visible instead of impossible — the confirmation design most agents need and few implement.
  • Published coverage limits. GitLab documents that its pre-receive secret check is skipped when a file or diff exceeds 1 MiB, or when a push changes more than 3,150 paths or 350,000 lines, and emits an audit event when it skips. Reading that list feels like a weakness disclosure. It is the opposite: a stated boundary is monitorable, and an unstated one has the same hole with no alert. We compared all four tools, and their bypasses, in GitHub Secret Protection vs GitLab push protection vs TruffleHog vs Gitleaks.

Translate each into an agent surface and the pattern is immediate: a result state for “the checker errored”; an override that is available and recorded rather than hidden; and a visible statement of what the agent's verification does not cover, shown where the user decides whether to look.

Step 5Spend friction where the measured gap is, not where the consequence class is.

Confirmation steps, diffs-before-apply and “show your work” are calibration devices before they are safety mechanisms. Friction on a high-consequence, low-track-record action makes the user look, which is the moment calibration happens. Friction on a high-track-record, low-consequence action is pure tax, and worse than neutral: it trains the user to click through without reading, and that habit transfers to the actions that mattered.

The usual rule — allocate friction proportionally to consequence — is only half right, and STEP 2 shows why. Consequence tells you how much a mistake costs; the measured per-band gap tells you where the mistakes are. A high-consequence action the agent has performed 4,000 times with two corrections needs a record, not a dialog. A medium-consequence action in the band where displayed confidence overshoots observed accuracy by 26 points needs the dialog, however routine it looks.

Track record is the instrument that makes this legible to users, and it needs a denominator: “drafted 240 replies, 6 edited by you, 0 reverted” is something a person can reason about; “highly accurate” is not. Scope it to the current task class and show it where the user is deciding how much to check — a global accuracy figure averaged over everything hides exactly the per-domain variance that calibration needs.

Step 6Do not display a signal you have not scored.

An unvalidated confidence number does not reduce over-trust; it concentrates it exactly where nobody is checking. That is the one rule on this page with no exceptions, and it has a concrete test attached: if items badged with your highest confidence are wrong more often than items in the band below, the badge is manufacturing over-trust and should be removed today, before it is improved.

Where a signal does survive scoring, render it as a routing input rather than an ornament — a number on every output is noise that teaches users to ignore the channel:

# Confidence is a routing input, not a UI ornament.
# Three states, because "we did not check" is not "we checked and it passed".
if check.ran and check.passed:
    render(answer, badge="checked", evidence=check.citation)
elif check.ran and not check.passed:
    render(answer, badge="check-failed", expand_reasoning=True)
else:
    # No checker for this claim type, or the checker errored.
    render(answer, badge="not-checked", why=check.unavailable_reason)

The measurement that licenses any of this is a production calibration report, and it is a day of work rather than a platform: join last month's outputs to whatever outcome you have, bucket by displayed confidence, and print band, n, accuracy and signed gap. The label sources, why Expected Calibration Error is the wrong number to track, and what a calibration monitor should page on are in measuring calibration in production.

Run the three-disclosure audit on your own surface this week. For each badge or confidence indicator, answer: what substrate backs it, what does the user see when the checker did not run, and where is the coverage boundary stated? A badge that cannot answer all three is a claim your product is making on the model's behalf — and when it is wrong, the user will be right to have believed it.