Prefill, decode & the KV cache.
One model call is two machines with opposite bottlenecks, and nearly every latency question you have dissolves the moment you separate them — why the first token takes a second while the rest arrive steadily, why doubling your prompt barely changes generation speed, why prompt caching is possible at all. Prefill reads the whole prompt at once and is limited by compute; decode writes the answer one token at a time and is limited by memory bandwidth.
Two phases, two bottlenecks.
Prefill pushes your entire prompt — system instructions, tool definitions, retrieved documents, conversation history — through the network in a single parallel pass. It produces the first output token and, more importantly, the internal state needed to produce all the rest. It is compute-bound: the GPU's arithmetic units are the constraint, and attention cost grows with the square of prompt length.
Decode then generates the answer one token at a time. Each step must read the full model weights and all the stored state from memory to produce a single token, which is then fed back in for the next step. It is memory-bandwidth-bound: the GPU spends most of its time moving bytes, not computing. It is also strictly sequential — token n+1 cannot begin before token n exists.
- Time to first token (TTFT) is essentially prefill. It scales with how much you sent.
- Inter-token latency is decode. It is roughly flat per token and barely notices how long your prompt was.
- Total wall-clock ≈ TTFT + (output tokens × inter-token latency). Two independent terms, two different fixes.
"The model is slow" is not a diagnosis. A request that waits two seconds and then streams smoothly and a request that starts instantly and dribbles for thirty seconds have nothing in common except the total. One is a context problem; the other is an output-length problem. No model swap reliably fixes both.
The KV cache is the state between tokens.
Attention requires every token to look at every earlier token — specifically at their keys and values, two vectors each layer computes per token (see transformers). Recomputing those for the whole history at every generation step would be ruinous, so they are computed once and kept. That store is the KV cache, and prefill's real product is a fully populated one.
- It grows linearly with sequence length — every token added, whether yours or the model's, adds a permanent row for the rest of the request.
- It is per-request. Ten concurrent conversations mean ten caches resident in GPU memory simultaneously. Serving many users is largely a KV-cache capacity problem, not a weights problem.
- It explains a common local-model surprise. The weights fit in your VRAM and the model still runs out of memory at long context, because the cache is what overflowed — which is why small local models are often constrained by context, not by parameter count.
- Long context has a memory price, not only a token price. This is the physical reason a 200K-token window is not simply a bigger number in a config file.
What this one split explains.
Several behaviours that look unrelated are the same fact viewed from different angles:
- Why prompt caching is a prefix match. Caching a prompt means storing the KV cache that prefill produced and reusing it next time. Because a token's keys and values depend on every token before it, the reuse must start at position zero and stop at the first difference — one changed byte early invalidates everything after it. That constraint is not a vendor policy; it falls straight out of the mechanism.
- Why output tokens cost several times more than input tokens. Input is processed in one amortised parallel pass. Each output token is its own pass over the entire model. The price difference is the work difference.
- Why a server batching many users gets throughput almost for free. Decode is bandwidth-bound, so weights loaded once can advance many sequences in the same step. This is also why your own single-stream latency does not improve when the provider adds capacity.
- Why reasoning models feel slow out of proportion to their answers. Thinking tokens are decode tokens, generated one at a time at full cost, and most are never shown — see reasoning models.
- Why streaming works at all. Tokens genuinely become available one by one; streaming is exposing the decode loop rather than simulating progress.
Optimising the half that is actually slow.
Once you can name the phase, the interventions stop competing:
- Prefill-heavy — long retrieved context, short answer. Cache the stable prefix, cut retrieved documents, keep volatile content at the end so the cacheable region stays intact. Sending less is the only thing that reliably helps.
- Decode-heavy — long generation, reasoning, verbose formats. Ask for less output: a schema instead of prose, a table instead of a narrative, a lower thinking budget. Output length is a latency dial that most teams never touch.
- Throughput-limited — many concurrent users on fixed hardware. Cap per-request context, because concurrency is bounded by KV memory, and every long-context request you allow costs you several short ones.
- Neither — if the wait is queueing rather than either phase, none of the above matters and the answer is capacity.
Before arguing about a faster model, measure TTFT and inter-token latency separately on one real request. If TTFT dominates, your project is context and caching, and a model swap will disappoint. If generation dominates, your project is asking for fewer output tokens. It is one measurement, and it points at two completely different pieces of work — which is why teams that skip it so often optimise the half that was already fast.
Related: context windows for the budget this all runs inside, tokens for what is being counted, and cost, quality & latency for the trade-off these mechanics produce.