Constrained Decoding

What the mask actually does.

Constrained decoding compiles your schema or grammar into an automaton and runs it alongside generation. At every decode step the engine walks the automaton to work out which tokens could still lead to a valid string, sets the logits of every other token to negative infinity, and samples from what remains. Validity stops being something the model achieves and becomes something the sampler cannot violate — see prefill & decode for where in the loop this happens, and temperature & sampling for what the mask is applied on top of.

• The cost is no longer the objection. XGrammar — the default structured-generation backend in vLLM, SGLang and TensorRT-LLM — reports per-token mask overhead under 40 microseconds, which disappears next to the forward pass. "Constrained output is too slow" was a 2024 argument.
• Two engine families, different ceilings. FSM-based engines (Outlines) cover regular languages and either reject a recursive schema or flatten it to a fixed depth. CFG-based engines (XGrammar, llguidance) handle genuine recursion. If your schema contains a tree — a nested expression, a comment thread, a folder — you need the second kind.
• "Structured outputs" is a marketing term, not a mechanism. One vendor's flag means true token masking; another's means generate-then-validate-then-retry; a third's means nothing more than an instruction plus a JSON-ish decoding mode. Those give you different guarantees and very different tail behaviour. Check which one you bought — JSON Schema subsets per vendor has the comparison.

Syntax is free. Semantics is not.

The mask guarantees exactly one thing: the bytes will parse. It says nothing about whether the values are true, and it removes the model's most honest failure mode. An unconstrained model that has nothing to say can trail off, hedge, or answer in prose. A constrained model must emit the next legal token, and the next legal token is never silence.

• Forced-field fabrication. A required invoice_total on a document that has no total is not left blank — the automaton demands a number, so a number appears. This is the single most common way constrained pipelines produce confident nonsense, and it looks identical to success in every dashboard that counts parse rate.
• Enum coercion. Given a closed label set that does not contain the right answer, the model picks the nearest legal member. You do not get an error; you get a plausible wrong category, with the "none of these" signal masked out of existence.
• Early closing. When the automaton allows }, a model running low on budget or confidence will take it. Arrays come back with two of the seven items and no indication that five are missing.

The fix is not to abandon constraints — it is to put the escape hatch inside the schema. Nullable fields for anything the source may not contain, an explicit "unknown" member in every enum, and a top-level branch the model can take to decline. If the schema has no way to say no, the model has no way to say no, and you have engineered abstention out of your system. See hallucination & grounding.

Where the "does it hurt quality" argument actually lives.

There is a real mechanism behind the worry. Masking can force a non-canonical tokenization — a byte sequence the tokenizer would normally have merged differently — so the model finds itself continuing a token path it almost never saw in training. That is a genuine distribution shift, and it is why the question keeps coming back. See tokens & tokenization.

The empirical picture is contested. The EMNLP 2024 study Let Me Speak Freely? reported that format restrictions hurt reasoning tasks while helping classification; dottxt's Say What You Mean could not reproduce it on the same model and attributed the gap to prompt and parsing artifacts rather than to the mask. Both can be read honestly, and neither settles it — which is exactly why you should not design around the disputed part.

Design around the undisputed part instead: a schema that starts at token one takes away the scratchpad. Whatever the mask does or does not cost, forcing the first token of the response to be { means the model must commit to an answer before it has generated any of the intermediate text it would otherwise use to get there. Separate the channels:

• Reason unconstrained, then constrain. A reasoning model's thinking channel is not schema-bound, and shouldn't be. Apply the grammar to the final answer only.
• Or run two passes. A free-form call that solves the problem, then a cheap, small-model extraction call under constraint that turns the prose into the object. Two calls, and the expensive one is unconstrained.
• Or keep the reasoning inside the schema, in the right order — which is the next step.

Design the schema for the decoder, not for your database.

Under constrained decoding the schema is not a validation contract that runs after the fact — it is the generation plan. Field order is generation order, nesting is branching, and every keyword you add is a place the automaton can force the model's hand.

• Order fields so evidence precedes conclusions. {"quote": …, "reason": …, "label": …} lets the model condition its label on text it has already written. {"label": …, "reason": …} makes the reason a post-hoc rationalisation of a label chosen with nothing behind it. This is the cheapest accuracy improvement on this page and it costs one line of schema.
• Flatter and shorter. Every nesting level and every verbose field name is tokens you pay for on a path the model did not choose. Reshape into your storage model afterwards, in code.
• Enums for closed sets — plus the escape member. A closed set is where masking earns the most, and where an omitted "none of these" does the most damage.
• Stay inside the supported subset. Strict modes accept a subset of JSON Schema; unsupported keywords are variously rejected loudly or dropped silently, and the silent case gives you a constraint you believe in but do not have. Structured outputs vs tool calls covers when to use which surface.
• Measure two rates, not one. Parse rate will be 100% and tells you nothing. Track field-level accuracy, and separately the abstention rate on a set of inputs where the correct output is "no answer".