Speculative Decoding

Draft, verify, keep the prefix.

Generation is slow for a boring reason. During decode, producing one token requires reading every weight in the model out of memory, and a modern GPU can read memory far more slowly than it can multiply. One token per full sweep of a 70-billion-parameter model is a terrible use of the hardware: the arithmetic units sit idle waiting on bandwidth.

Speculative decoding exploits that idle compute. Each round has two halves:

• Draft. Something cheap proposes the next k tokens — typically 3 to 8. That "something" can be a small model from the same family, a lightweight head bolted onto the big model's own hidden states (EAGLE, Medusa), or, in the cheapest case, an n-gram lookup that just copies a phrase already present in the prompt.
• Verify. The full model runs once over the whole proposed span, scoring all k positions in parallel. Because the sweep of weights is what costs, scoring eight positions costs barely more than scoring one.

Then the accept rule: walk the draft left to right, keep each token while it agrees with what the full model would have sampled, and cut at the first disagreement — replacing that token with the full model's own choice. A round that accepts five of six drafted tokens has produced six tokens for the price of roughly one forward pass.

Acceptance rate is the whole equation.

Two numbers decide whether any of this helps: how often the draft is right, and what the draft cost to produce. Everything else is detail.

• Acceptance rate is the fraction of drafted tokens that survive verification. It is not a property of the technique — it is a property of the draft/target pair on your traffic. Well-matched modern draft heads report roughly 0.8 on code and instruction-following; a mismatched pair on unusual text can fall by half.
• Draft cost is what you spend generating a proposal you may throw away. A separate small model that is a tenth the size costs a real tenth of a forward pass per drafted token; a head trained against the big model's own hidden states costs far less and is why that design took over.

The two combine in an obvious way and an unobvious one. Obviously, raising k raises the ceiling on tokens-per-round. Unobviously, acceptance compounds: a token is only usable if every token before it in the same round was also accepted, so expected accepted length grows sub-linearly and each extra speculative token is worth less than the one before it. Past a certain k you are paying to draft tokens that are almost never reached.

Two practical consequences fall out of this:

• Acceptance is content-dependent, and predictably so. Highly patterned output — code, structured JSON, a document being lightly edited, anything where the next span is nearly copyable from context — accepts far better than open-ended prose. This is why the trivially cheap n-gram drafter is genuinely competitive on structured output and on edit-a-file workloads.
• Acceptance is worth logging. It is the one number that tells you whether the feature is still earning its keep after a model swap, a prompt rewrite, or a shift in what your users ask for. A drop in acceptance is the leading indicator that the draft has stopped matching the traffic.

The trade is latency for compute — so load decides.

This is the part that surprises people, and it is the reason a technique advertised as a 2–3× speedup sometimes makes a production system slower.

Speculation works by converting spare arithmetic into fewer sequential steps. On a GPU serving one request, there is enormous spare arithmetic and the trade is close to free. But a serving engine's answer to an idle GPU is batching: run many requests through the same weight sweep at once. A large batch already fills the arithmetic units, and at that point the machine is compute-bound rather than bandwidth-bound — there is no idle capacity left to spend, and every rejected draft token is compute that could have served another user.

• Low concurrency — interactive chat, a single agent stepping through a task, anything where a user is watching tokens appear. Speculation is close to pure win: you were wasting the compute anyway.
• High concurrency — a saturated endpoint serving many requests. Speculation can reduce total throughput, and with it your tokens-per-dollar. Serving engines expose this as a load threshold for exactly this reason; vLLM's --speculative-disable-by-batch-size turns speculation off once concurrent requests exceed a limit you set.
• Long context shifts the line, because a large KV cache pushes the system back toward bandwidth-bound even at higher batch sizes. Where your own crossover sits is a measurement, not a number you can look up.

Where it sits among the other speed levers.

Put speculation next to its alternatives and the shape of the decision is clear. Only one of these columns is free of a quality argument.

• Quantization shrinks the weights so each sweep reads less memory. It is the bigger single-stream win and it stacks with speculation — but it changes outputs, so it costs you an eval cycle. See distillation and quantization.
• A smaller model is the largest win of all and the largest quality risk; routing the easy requests to it is usually smarter than serving everything from it.
• Prompt caching attacks prefill rather than decode, which is often where the perceived latency actually lives on long-context agent turns.
• Streaming does not make anything faster, it makes the wait legible — and on most interfaces that is worth more than a 20% decode improvement. See streaming and partial output.

If you buy inference from an API rather than running it, none of this is a knob you turn — it is already applied on your behalf, silently, and it is part of why provider latency differs on identical models. It becomes your decision the moment you self-host.

Related: prefill and decode for the two phases this optimises between, cost, quality & latency for the trade-off it sits inside, and batch and async inference for the workloads where latency is not the metric at all.