Thresholds for Decision Models

Step 1Why the threshold is the artifact, and why nobody hands it to you.

A decision model — TypeSafe's Jev, the OpenAI Decisions API, Cloudflare's Clef and the rest of the wave compared in the category write-up — takes a state and a set of typed questions and returns a probability distribution instead of prose. That removes malformed output. It does not remove wrong output, and it does not tell you at what probability an answer is safe to act on. That number is yours to find, and it is the part of the integration that decides whether it works.

The evidence you would want is missing. As of October 2026, TypeSafe's launch post claims "higher confidence means higher accuracy" with no metrics, and its "antibenchmaxxing" post says it will not publish public benchmark tables at all. OpenAI's Decisions guide tells you to set thresholds from labelled examples and publishes no calibration numbers either. The only published expected calibration error for Jev comes from a third party, the Pinocchio paper (arXiv:2609.24881): ECE 0.133, AUROC 0.666, Brier 0.240 over 1,718 responses — on an off-label task, grading another model's answers, not the in-distribution classification Jev is sold for. It is a warning, not a number you can use.

One more thing narrows the signal: Jev's confidence field is a deterministic function of probabilities. For a Choice it is (n·p_max − 1)/(n − 1), so (0.6, 0.3, 0.1) and (0.6, 0.2, 0.2) both read 0.4. It adds no information the distribution did not already carry; any calibration claim attaches to the probabilities. The general theory of turning a score into an abstention gate lives on uncertainty and calibration; this page is the runbook for doing it against a decision-model API.

Step 2Build the labelled set from your own traffic, and size it by arithmetic.

Run the decision model in shadow beside the LLM call it is meant to replace, on real production inputs, and record both answers plus the full probabilities object. A curated eval set will not do: calibration is a property of a distribution, and the only distribution that matters is yours. TypeSafe's MIT-licensed system-one-adapter-python runs the same typed questions against OpenAI, Anthropic, Gemini or any OpenAI-compatible endpoint, so the incumbent and the candidate can answer the identical question shape — the natural harness for the comparison.

Label ground truth with humans, not with the incumbent LLM. Agreement with the old model measures imitation; you want correctness. If you scale labelling with a model judge, calibrate the judge first per LLM-as-judge for agents.

Size the set from the error rate you need to see, not from a round number. Suppose the auto-act band must hold errors near 2%, and you want to know that to within ±1 point at 95% confidence. The normal approximation gives n = 1.96² × 0.02 × 0.98 / 0.01² ≈ 753 labelled examples inside the act band. If you expect that band to hold about half your traffic, that is roughly 1,500 labelled inputs overall, before you have looked at the bands below it. The zero-error shortcut is the same message from the other side: seeing no errors in n examples only bounds the true rate at about 3/n, so 300 clean examples prove "under 1%", not "zero". Tighter tolerance costs quadratically more labels; decide what you can afford to know before you pick a tolerance you cannot verify.

Step 3Read calibration off the probabilities, then pick the operating point.

With the labelled set in hand, take the top probability of each answer (for a Noul, the probability itself; it has no confidence field, and a value near 0.5 is the model saying it is unsure). Bin it — tenths are fine at this sample size — and plot accuracy per bin against the bin's mean probability. That reliability diagram tells you whether "0.9" means nine in ten on your traffic. Then sweep a threshold and draw the coverage–risk curve: at each cut, the share of inputs you would answer and the error rate among them.

Choose the threshold from the curve, per question, by the error you can tolerate for that question. Routing a ticket to the wrong queue and auto-approving a refund are different questions with different thresholds; a single global threshold means you set it for the riskiest one and pay in coverage everywhere else. TypeSafe's docs suggest also trying the top probability and the ratio of top to second probability as signals; if a confirm threshold sits low, a margin test (top minus second) separates a clear 0.55 winner from a 0.55/0.45 coin-flip.

Step 4Three bands, scaled to blast radius, with somewhere real to escalate.

TypeSafe's confidence docs recommend three bands rather than one cut: high confidence acts, medium confirms, low escalates, with thresholds scaled to the risk of the action. Use that shape.

• Act. Above the upper threshold the decision executes with no review. This is the only band that saves money, and the only one whose error rate you measured precisely in STEP 2.
• Confirm. Between the thresholds the decision is prepared and a cheap check — a human click, a deterministic rule, a second question — releases it.
• Escalate. Below the lower threshold the input goes to the old LLM path or to a person. Keep that path alive and budgeted; decommissioning it the week the new model lands is how the escalate band becomes an outage.

Scale the upper threshold to what an error costs. A reversible label can live with a lower bar than an irreversible action, and an action that touches money or a customer should usually have no act band at all until the data says otherwise.

Step 5Probe the known jagged edges before trusting the curve.

A clean coverage–risk curve on in-distribution traffic says nothing about the ways the model is known to bend. Test each before you promote:

• Option order. TypeSafe's Jev 1.13 jaggedness page (last reviewed 2 October 2026) says a Choice "leans toward the option that comes first". Re-run a sample with options permuted and measure how often the answer or the band changes.
• Repeat requests. Pinocchio reports Jev returning different scores for identical repeated requests, worth roughly 0.02 of run-to-run variation in their AUROC; TypeSafe's docs promise similar answers for similar inputs but no determinism guarantee. Send identical requests several times and count band flips near each threshold.
• Adversarial state. The docs concede the model does not treat state as hostile. HiddenLayer showed an injected page element hijacking a checkout decision; Wu & Lim (arXiv:2609.28613) found injected text shifts probabilities but rarely flips the chosen answer, with adaptive attacks raising success from 1.8% to 3.5%. Read that carefully: a shifted probability can move an answer across your threshold even when the argmax holds.
• Arithmetic and dates. The docs say plainly that Jev "is not a calculator" and reads dates as text. Compute counts, sums and date comparisons in code and ask the model only the judgement that remains.

Step 6Pin the version, tie thresholds to it, and watch the bands.

A threshold is a fact about one model version. The alias moves underneath you: as of 19 October 2026 jev-latest points to 1.13.0, and TypeSafe's 1.14 post says it moves to 1.14.0 one week after the 14 October release, so around 21 October. The docs say aliases move silently and advise pinning the versioned ID if you tuned thresholds against one version. TypeSafe plans 60 days' notice before 1.13.0 is deprecated and 30 days for 1.14.0; the formal deprecation policy is promised but unpublished. Treat every bump as a new model: re-run the labelled set, redraw the curve, re-derive the thresholds, per model deprecation and migration and unpinned vendor defaults.

# request: pin the version the thresholds were derived on
{ "model": "jev-1.13.0", ... }   # not "jev-latest": the alias moves silently

# thresholds are keyed by version; values here are illustrative
THRESHOLDS = { "jev-1.13.0": { "act": 0.97, "confirm": 0.80 } }

# choice/score answers; for a noul, use max(p, 1 - p) of its "noul" value
def band(response, question):
    t = THRESHOLDS[response["model"]]       # unknown version -> KeyError, by design
    probs = response["answers"][question]["probabilities"]
    top = max(probs.values())                  # read probabilities, not "confidence"
    if top >= t["act"]:     return "act"
    if top >= t["confirm"]: return "confirm"
    return "escalate"                         # old LLM path or a human

The response echoes the model that answered, so key the lookup on it rather than on what you asked for; an unknown version then fails loudly instead of silently borrowing old thresholds.

In production, three signals carry most of the weight. Band populations: the share of traffic in act, confirm and escalate is the cheapest regression signal you have, because it needs no labels and moves the day the model or the input distribution shifts. Escalation rate: it is the real cost driver, since every escalated input pays for both models. Spot labels from the act band: sample a small, steady slice of auto-acted decisions for human review, because that band is where errors ship unseen and where your STEP 2 estimate quietly goes stale. Wire all three into quality regression detection.