Structured outputs.
Turning on schema enforcement deletes the failure you were testing for and installs three you are not: a refusal that comes back in a shape your schema never described, a truncation that yields a valid prefix of an invalid document, and a rejection that fires before a single token is generated. All three live outside the schema, so the typed client you just generated handles none of them. Structured output is worth every bit of this — it is the thing that makes a model composable with software — but the work moves from coaxing the format to designing the four-branch parse path and the schema that decides how often the content is true.
Step 1Prose does not compose, and asking nicely is not enforcement.
Models are trained to produce fluent natural language, which is exactly wrong when the consumer is software. If you classify a ticket and the model replies "This looks like a billing issue, probably high priority since they mentioned a failed payment," your code has to parse intent out of prose — brittle, and it breaks the day the model phrases it differently.
UNUSABLE "This looks like a billing issue, probably high priority." USABLE { "category": "billing", "priority": "high" }
Asking for JSON in the prompt gets you most of the way, and "most of the way" is the problem: the model still samples token by token, so it occasionally wraps the object in ```json fences, adds a preamble, emits a trailing comma, invents a key, or returns a number as a string. At a 1% failure rate across 100,000 daily calls that is 1,000 broken records a day.
The ladder out of that is short and well worn — prompt and few-shot, then a JSON mode that guarantees syntactic validity but not your schema, then schema-constrained decoding, which restricts the sampler at each step to tokens that keep the output a valid prefix of something the schema allows. A tool call's argument object is the same machinery pointed at a function instead of at your application; see tool calling and, for how the mask is built, constrained decoding. As of October 2026 the top of that ladder is the default rather than a frontier feature — it is generally available across the major APIs, on the Claude API as an output_config.format of type json_schema plus a strict flag on tool definitions. So the interesting question stopped being "can I get valid JSON" and became "what exactly did that guarantee cover."
Step 2What the guarantee covers: structure only, and optionality is not free.
Constrained decoding guarantees your output is a syntactically valid instance of the schema's shape. It does not enforce the parts of JSON Schema that describe content. Across the strict modes, numerical bounds (minimum, maximum, multipleOf) and string-length bounds (minLength, maxLength) are not enforced, objects are generally required to set additionalProperties: false, and recursive schemas and some regex constructs are unsupported. A schema that says "an integer between 1 and 5" reliably gets you an integer and tells you nothing about the 5.
The second restriction is the one that changes how you design, because it is counterintuitive: in strict mode a field is not optional in the sense you mean.
- OpenAI's strict mode has no optional fields at all. Its own guide is explicit that to use Structured Outputs every field or function parameter must be listed as required, and that the way to express an optional value is a union with
null. The model returns a value for every parameter; you get a key whose value may benull, not an absent key. - Anthropic allows genuine optionality but rations it. Its documented limits apply per request, across every strict schema in that request: at most 20 strict tools, 24 optional parameters in total, and 16 parameters carrying union types. Schema compilation has a 180-second timeout, and an unsupported feature returns a 400 with details rather than silently degrading.
Put those together and the design consequence is sharp. Every field you declare in a strict schema is a question the model is forced to answer on every single call — either because optionality is unavailable and the only escape is an explicit null you must legislate for, or because optionality is a metered resource you will run out of. A codegen pipeline that emits forty "just in case" fields is not being thorough; it is issuing forty compulsory questions per request and, on Anthropic, failing the 24-optional cap outright.
Which keywords survive which vendor, and what ports between them, is its own subject — JSON Schema subsets per vendor maps the three dialects, and structured outputs vs tool calls covers choosing between the two doorways.
Step 3The three failures that live outside the schema.
Here is the trade nobody prices. Before enforcement you had one failure mode, and it was loud: the text did not parse. You caught it with a try block. After enforcement that failure is gone, and three others matter more — and none of them is schema-shaped, so a generated typed client walks straight into all three.
- The refusal arrives in a different shape. If the model declines, it cannot express that inside your schema — there is no field for "I won't." Providers return it out of band, as a separate refusal field or content object. Code that reaches for the parsed object sees a missing or empty structure and typically reports a parse error, which sends you debugging the schema instead of reading the reason. See refusals and capability gating.
- Truncation produces a valid prefix of an invalid document. Constrained decoding guarantees that what has been emitted so far could be completed validly; it cannot guarantee the model gets to finish. Hit
max_tokensmid-object and you have well-formed JSON up to the cut and nothing after it. This is the top cause of broken structured output once enforcement is on, and it is a budgeting bug rather than a format bug — the same boundary streaming and partial output deals with. - The schema itself is rejected, before generation. An unsupported keyword, a recursive
$ref, one optional field past the cap, or a grammar too expensive to compile inside the timeout — all of these fail the request outright. This failure has nothing to do with the model's behaviour and everything to do with your schema, so it is deterministic: it will fail identically forever. Retry logic that treats a 400 as a transient model error will retry it forever, which is how a one-character schema change becomes a billing incident.
So the parse path has four branches, not two, and writing them out is most of the work:
response = call(model, schema, max_tokens) 1. REFUSED -> refusal field present surface the reason; do NOT retry blind 2. TRUNCATED -> stop_reason == max_tokens retry with a larger cap or a smaller schema; the output is a fragment 3. SCHEMA 400 -> request rejected pre-generation DETERMINISTIC. fail fast, page a human, never put this in a retry loop 4. OK -> parse, then validate SEMANTICS in code # only branch 4 is the one the schema was about.
Catch the schema rejection in CI, not in production. Your schemas are static artefacts: send each one to the provider once in a build step with a trivial prompt and assert it compiles. That turns branch 3 from a runtime outage into a failed test, and it is the single highest-value thing on this page — the failure is deterministic, so a test catches it with certainty.
Step 4Because every field is compulsory, the schema is your fabrication lever.
Enforcement guarantees the shape and never the truth: {"priority": "high"} is perfectly valid and may be wrong. What STEP 2 adds is that you cannot reduce fabrication by leaving a field out and hoping — in strict mode it will be filled. The schema is therefore the main control you have over how often the content is right, and four decisions carry most of it.
- Always legislate an escape hatch. This is now the first rule rather than a nicety. With no nullable branch and no
"unknown"enum member, a forced schema forces a fabricated value — the model has no legal way to decline a field. Give it one and confabulation drops; this is the structured-output face of calibration, and of unsatisfiable tasks generally. - Order the fields so reasoning precedes the verdict. Generation is left to right, so a
decisionfield placed before itsjustificationis a guess the model then rationalises. Put the short reasoning field first. This is the one piece of schema design with a literature behind it, and it is why the "does enforcement hurt quality" debate resolves the way it does — STEP 5. - Enums over free strings.
"low" | "med" | "high"cannot drift to "kinda urgent." The enum is enforced while apatternon a free string largely is not, so it does real work the regex does not. - Keep it shallow, named and small. Field names act as inline instructions, deep nesting degrades content quality even where syntax is guaranteed, and every field is a compulsory question — narrow the schema to what the model has business filling, the discipline schemas and contracts argues for on the tool side.
FORCES A GUESS -- 6 compulsory answers, no escape { "category": enum[billing|technical|account], "priority": enum[low|med|high], "customer_sentiment": enum[happy|neutral|angry], "root_cause": string, "estimated_fix_minutes": integer, "decision": enum[refund|escalate|close] } ASKS WHAT IT CAN ANSWER -- reasoning first, outs everywhere { "evidence": string, "category": enum[billing|technical|account|unclear], "priority": enum[low|med|high|unclear], "decision": enum[refund|escalate|close|needs_human] } # estimated_fix_minutes had no basis in the ticket, so it # was a compulsory invention on every single call.
Step 5Does enforcement hurt quality? The ordering does, not the constraint.
This argument has a real history and it is worth knowing which way it landed, because the folklore is a version ahead of the evidence. The 2024 study Let Me Speak Freely? (EMNLP Industry) reported that format restrictions degrade reasoning while helping classification, with JSON mode the worst performer on reasoning tasks — and that result is still widely quoted as "constrained decoding makes models dumber." Later work complicates it in a useful direction: a 2026 paper re-running the comparison under matched prompts and output formats found constrained decoding consistently matching or beating unconstrained, while still noting that reasoning inside JSON underperforms reasoning in free text. A separate 2026 preprint attributes the original effect to premature serialisation — the model being made to emit schema-compliant tokens before it has finished thinking.
That is an actionable conclusion rather than a wash: the penalty attaches to where the reasoning goes, not to the mask. So keep enforcement and give the reasoning somewhere to live — a first string field inside the schema, or a separate unconstrained call whose output feeds a constrained extraction pass. What you do not do is put a bare verdict field at the top of a strict schema and then conclude the model cannot reason.
Then stay defensive regardless of enforcement:
- Reserve output tokens. Size
max_tokensfor the largest valid object — including the reasoning field you just added, which is the usual cause of a truncation that appeared the day someone improved the schema. - Validate semantics in code. The schema cannot check that an email exists, an ID resolves, or a total adds up, and with bounds keywords unenforced, range checks are your job.
- Parse-and-retry, with a budget and a reason. Feed the validation error back and ask for a fix; one retry recovers most failures. Branch on why it failed, per STEP 3, and never retry branch 3.
- Count the escape hatches you get. If
"unclear"never appears in production, either your inputs are cleaner than you think or the model learned the hatch is unwelcome — the rate is a monitoring signal. See agent evaluation.
Turn strict mode on — it is free and it deletes a real failure class. Then spend the afternoon it bought you on the three failures it does not cover: add a CI step that compiles every schema against the provider so a rejection is a failed test rather than a 400 in a retry loop; branch your parse path on refusal, truncation and schema-rejection separately before you reach for the object; and audit each field for whether the model can actually answer it from the input, because strict mode will make it answer anyway. If you change one thing, add "unknown" to every enum and a reasoning field above every verdict.
Deliverable
You know prose does not compose, and that schema-constrained decoding excludes invalid tokens at generation time rather than cleaning up afterwards. You know what the guarantee covers as of October 2026 — structure, not content, with numeric and string bounds unenforced — and that strict mode either bans optional fields outright or rations them, so every field you declare is a compulsory question. You can name the three failures that live outside the schema (refusal, truncation, and the deterministic pre-generation rejection that must never enter a retry loop), you write a four-branch parse path, and you compile your schemas in CI. And you design against forced fabrication with explicit unknowns and reasoning placed before verdicts — which is also the honest answer to whether enforcement hurts quality, since the penalty attaches to premature serialisation rather than to the mask.