Outlines vs XGrammar vs llguidance vs Instructor: Valid JSON Was Never the Hard Part

At a glance

Four projects that all appear under "structured outputs" and do not all do the same thing.

How constrained decoding actually works

The mechanism is simpler than the project count suggests. At every decoding step the model produces a distribution over the whole vocabulary. A grammar engine computes, for the current parser state, which tokens could legally come next; every other token has its logit set to negative infinity before sampling. Invalid output is not corrected, it is unreachable — and because the mask applies before sampling, the guarantee holds at any temperature.

That leaves exactly one hard engineering problem: computing the mask fast enough that it does not dominate the step. A vocabulary is 100,000-plus tokens and a step is single-digit milliseconds, so the mask has to be produced in microseconds. Every difference between the three engines is a different answer to that problem, and every one of those answers is a trade between work done up front and work done per token.

The counter-intuitive result

Constrained generation is often assumed to cost latency. Benchmarks across large collections of real-world JSON schemas have repeatedly found the opposite in aggregate: a well-implemented engine can produce lower per-token latency than unconstrained generation. The reason is not magic — a constrained model finishes sooner. It cannot ramble, cannot emit a preamble, cannot re-open a closed object, and frequently there is only one legal token, which some engines fast-path without consulting the model at all. If your objection to structured decoding is that it will slow you down, measure before you believe it.

The three grammar engines, and the one question that separates them

Outlines — compile once, then it is nearly free

Outlines popularised the approach: turn the schema into a finite automaton and precompute, for every state, the set of allowed tokens. The result is a lookup at sampling time, which is about as cheap as this can get. The costs are the mirror image — building that index takes real time and memory for a schema you have not seen before, and a finite automaton cannot express unbounded recursion, so deeply recursive or self-referential schemas are either rejected or flattened to a fixed depth. If you serve a small fixed set of schemas, both costs are paid once at startup and never again.

XGrammar — just in time, with a cache that does the work

XGrammar reframes the problem around a context-free grammar and a compilation step that is fast enough to run on demand, backed by a persistent cache. Repeated schemas hit the cache and cost nothing; genuinely novel ones pay a compile that is small relative to a generation. It also handles the recursive schemas that trip an automaton-based approach, and its published overhead per token is low enough to disappear into normal step time. This combination is why it became the default structured-output backend in vLLM and SGLang rather than an option you opt into, and the 2026 follow-on work has been aimed squarely at the dynamic-schema case where agents live.

llguidance — pay nothing up front

llguidance, the engine underneath Microsoft's Guidance, takes the opposite position: build the automata lazily and compute masks on the fly, so there is essentially no startup cost at all. Under a fixed schema set this loses to a precomputed index on raw per-token throughput. Under genuinely unique-per-request schemas it wins, because the competition is paying a compile that llguidance never incurs. Independent comparisons on serving engines show exactly this crossover — XGrammar ahead on repeated simple schemas, llguidance ahead when every request brings something new.

So: do your schemas repeat?

That is the decision. A product with twenty fixed extraction schemas has near-total cache hits and should take whatever its serving engine defaults to, which today means XGrammar and means not thinking about it. A platform where tenants supply their own schemas, or an agent whose tool set is assembled per session, has a cache-miss workload and should benchmark llguidance against it. Outlines remains the reference implementation of the precompute idea and the right pick when you control the schema set completely and want the lowest possible per-token cost.

Instructor is not in the same category

Instructor does not touch the sampler. It wraps the provider client, sends your Pydantic model as a schema or tool definition, parses what comes back, validates it, and on failure sends the validation error to the model as a new request — by default up to three times. This is retry-based, not constraint-based, and the difference is not a matter of degree.

What that costs is obvious: a failure is a whole extra generation, latency is unbounded in the tail, and there is no guarantee at all — only an exception when the retries run out. What it buys is the part people undersell. A grammar can enforce that discount_percent is an integer between 0 and 100. It cannot enforce that discount_percent is zero unless customer_tier is "gold", that end_date is after start_date, or that every ID in the response appears in the retrieved context. Those are arbitrary predicates over a parsed object, they are the constraints that actually fail in production, and a Pydantic validator expresses all of them in a line.

The second thing it buys is reach. Grammar engines run inside a serving engine, which means you need to be running the model. Against a hosted API you get whatever structured-output feature the provider ships and nothing else. Instructor works everywhere, and where the provider does offer constrained decoding it uses it and keeps the validation layer on top.

The honest recommendation is both

These compose, and composing them is the configuration that actually works: a grammar engine guarantees the object parses, and a validation layer enforces the semantics the grammar cannot see. You then have exactly one retry path, and it triggers only on the errors that were always going to need one. Teams that pick a single tool usually end up rebuilding the other half by hand.

The failure mode nobody benchmarks

Schema compliance and correctness are different properties, and only one of them is measured by every benchmark in this space. A model forced into a schema will fill it. Ask for a required root_cause field and you will get one whether or not the evidence supports a root cause; ask for a five-value enum and the model picks one rather than telling you none applies. The constraint has converted "I do not know" — the most useful thing a model can say — into a confident, well-formed, unfalsifiable value.

• Give abstention a legal encoding. Make fields optional where the answer genuinely may not exist, and add an explicit insufficient_evidence variant to enums. If the schema has no way to express uncertainty, you have not eliminated uncertainty; you have hidden it.
• Do not constrain the reasoning. Forcing structure over the whole response constrains the tokens the model thinks in. Let it reason in free text, then emit the structured object as a separate constrained segment or a separate call.
• Field order is a prompt. Generation is left to right, so a field placed early is produced with less context and conditions everything after it. Put conclusions after evidence, not before.
• Names carry instruction. is_fraudulent and risk_flag are the same boolean and do not produce the same distribution. Schema key wording is a prompt-engineering surface, and recent work has measured it as one.

None of this is a criticism of the tools. It is the reminder that they solve the syntactic half of the problem completely and the semantic half not at all, and that the second half is where your incident reports come from.

When to pick which

One operational note that outranks the whole comparison: whichever engine you use, verify it still behaves at production batch sizes. Structured decoding is applied per sequence, and an engine that looks free at batch 1 can become visible when every request in a large batch carries its own grammar. That is an axis the serving-engine comparison treats as a first-class differentiator, and it is worth measuring on your own traffic rather than inheriting from a blog post.

Further reading

On this wiki:

• Structured outputs — the concept, from "ask for JSON" to schema-constrained decoding.
• Structured outputs vs tool calls — which mechanism to reach for.
• JSON Schema subsets per vendor — what each provider actually accepts.
• Streaming tool calls in practice — partial structured output on the wire.
• vLLM vs SGLang vs TensorRT-LLM vs llama.cpp — where these engines are hosted.

Project sources:

• Outlines
• XGrammar
• llguidance
• Instructor
• vLLM — structured outputs documentation
• Generating Structured Outputs from Language Models: Benchmark and Studies