KV Cache Budgeting for Agent Loops

The formula, and why it is the number that decides.

During prefill and decode, a transformer stores one key and one value vector per token per attention layer, so that generating token n + 1 does not require recomputing attention over tokens 1…n. That store is the KV cache, and its size is fully determined by geometry you can read off a config file:

# Bytes of KV cache for one sequence.
bytes = 2 * layers * kv_heads * head_dim * bytes_per_element * tokens
#        ^ one K and one V                 ^ 2 for fp16/bf16, 1 for int8

# Per-token cost is everything except `tokens`, so quote that:
bytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_element

Three properties of that expression are what make it the binding constraint for agents rather than for chat. It is linear in context length, and an agent loop appends tool results every step, so its cache grows with the step index and never shrinks unless you make it. It is multiplied by concurrency, because every in-flight sequence holds its own cache. And it is independent of how many parameters are active — a mixture-of-experts model that activates 9B of 74B parameters per token still pays full cache price on every attention layer it has, which is why MoE sparsity buys compute and not memory. The weight file is a fixed cost paid once; the cache is a variable cost paid per token per run, and only one of those two scales with the thing your agent does more of.

A worked budget on one 80 GB card.

Take a 70B-class dense transformer with the geometry that has become conventional for models in that range — 80 layers, grouped-query attention with 8 key-value heads, head dimension 128 — served at bf16 cache precision. These are assumptions, named so you can substitute your own model's config; the point is the shape, not the specific model.

bytes_per_token = 2 * 80 * 8 * 128 * 2 = 327,680 B = 0.3125 MB

Context    KV cache, one sequence
--------   ----------------------
   8,192                   2.5 GB
  32,768                  10.0 GB
 131,072                  40.0 GB
 262,144                  80.0 GB

Weights, 70B at 4-bit (~0.6 GB per billion)        42 GB
Runtime, activations, fragmentation (allow)         4 GB
--------------------------------------------------------
Headroom for KV cache on an 80 GB card             34 GB
  → one sequence at 32K context      10 GB     fits
  → one sequence at 128K context     40 GB     does not fit
  → three sequences at 32K context   30 GB     fits
  → four  sequences at 32K context   40 GB     does not fit

Read the last four lines as the actual deployment statement: this is a 32K-context, three-concurrent-agent server, not a "128K context" server, and no amount of advertised window changes that. The same card with the same model serves either one long agent or three short ones, and which of those you sell is a product decision you are making whether or not you notice you are making it. The context-budgeting discipline normally arrives as a cost and accuracy argument; on self-hosted hardware it arrives as a hard allocation.

Note also how badly the two halves of the budget respond to the obvious lever. Quantizing weights harder — 4-bit to 2-bit — saves 21 GB once. Doubling the context cap costs 10 GB per concurrent run, every run. Past a fairly early point, weight quantization is not where the memory is.

Four levers that change the per-token cost, and what each one charges you.

Every term in the formula except tokens is set by the model you picked, which makes the per-token figure a model-selection criterion rather than a tuning knob. These are the four things that move it.

• Fewer KV heads (GQA, MQA, MLA). The largest lever by far, because kv_heads enters linearly and multi-head attention sets it equal to the query-head count. NVIDIA's published config for Nemotron 3 Super is instructive: 32 query heads against 2 key-value heads, head dimension 128, 88 layers. Sixteen query heads share each KV head, which costs 16× less cache than full multi-head attention over the same geometry. The charge is a modest capability cost that model authors have judged worth paying for several generations; you are not choosing this at serving time, you are choosing it when you choose the model.
• Fewer attention layers (hybrid backbones). Only attention layers hold a cache that grows with length. State-space blocks — Mamba-2 and its relatives — carry a fixed-size recurrent state instead, so a hybrid backbone that interleaves Mamba, attention and MoE blocks pays length-dependent cache on only the attention minority. NVIDIA describes Nemotron 3 Super's language backbone as exactly that interleave. The charge is real and under-discussed: a fixed-size state is lossy by construction, so what you buy with the memory is a different forgetting profile, not free capacity. Read it alongside effective versus advertised context — a hybrid's advertised window and its usable recall can diverge for a structural reason, not a benchmark one.
• Lower cache precision. bytes_per_element is the one term you genuinely control at serving time: int8 KV halves the cache, int4 quarters it, and most serving stacks expose it as a flag. The charge shows up first on long-context retrieval, which is the workload that depends most on fine distinctions between cached keys — so measure it on your own long-context evals rather than on a short-prompt benchmark that cannot see the damage. Unlike weight quantization, which is covered in distillation and quantization, this one is cheap to try and cheap to revert.
• Paging and offload. Paged attention stops fragmentation from wasting the headroom you measured, and CPU or disk offload trades cache residency for bandwidth. Offload is the lever of last resort: it converts an out-of-memory error into a latency regression, which is sometimes the right trade and is never free. Prefix sharing is the honourable exception — many agent sessions sharing one system prompt and tool catalogue can share its cache blocks, which is the single best reason to keep that prefix byte-stable.

Working backwards from "fits in 24 GB".

Vendors increasingly ship a memory budget as the headline. NVIDIA's Nemotron 3.5 Bolt 74B-A9B-24GB, released on 15 October 2026, is compressed from the Super VL parent checkpoint with native 2-bit quantization to fit a 24 GB budget, and its model card states support for context up to 256K tokens. Those two numbers are the interesting part, because taken naively they do not fit together — and the way they are reconciled is exactly what a self-hoster needs to be able to check.

Do the division. 74B parameters at 2 bits is roughly 18.5 GB of weights. On a 24 GB card that leaves something under 4 GB for everything else, and a 256K cache has to live inside it:

Budget available for KV cache at 256K tokens:  < 4 GB
Implied per-token cost: 4 GB / 262,144 tokens  ≈ 16 KB/token

With kv_heads = 2 and head_dim = 128 at int8:
  bytes_per_token = 2 * L * 2 * 128 * 1 = 512 * L
  16,384 / 512  →  L ≈ 32 attention layers

At bf16 cache precision the same budget implies L ≈ 16.
Published backbone: 88 layers, interleaved Mamba-2 / attention / MoE.

So the claim is self-consistent only if a minority of those 88 layers are attention layers and the cache is held at reduced precision — which is precisely what a hybrid backbone with two KV heads is for. The arithmetic above is a reconstruction from the published budget, not a quote from a config file; read the real config before you plan capacity on it. But the method is the durable part: when a vendor publishes a memory number and a context number together, divide one by the other and ask what geometry the quotient implies. If the implied per-token cost is far below what a dense transformer of that size would charge, you are looking at a hybrid or an aggressive attention variant, and you should go and find out which, because the answer predicts how the model forgets — and whether its 256K is a window you can use or a window you can merely fill.

Capacity is a division, and the divisor is the step your agent has not reached yet.

Once the per-token cost is known, the serving question becomes arithmetic with no judgement left in it:

# How many concurrent agent sessions one accelerator serves.
sessions = (vram_total - weights - runtime_overhead)
         / (bytes_per_token * context_cap)

Two things about that denominator are easy to get wrong, and both of them are agent-specific. First, the right value for context_cap is not the average session length — it is the length you will allow, because memory must be reserved against the worst case or the twelfth session evicts the eleventh mid-run. An agent whose sessions average 20K tokens but are permitted to reach 200K is a 200K-token tenant for capacity purposes. Second, sessions are long-lived and overlapping in a way chat turns are not: an agent holds its cache across tool calls that may block on a network round trip, so utilisation and residency come apart, and a card that looks idle on a compute graph can be entirely full. Concurrency and scaling covers the queueing consequences, and multi-tenancy for agents the isolation ones.

The reserved-worst-case rule is what makes a context cap the load-bearing capacity control. Raising the cap from 32K to 64K on the worked example above does not cost a little more memory — it halves the number of agents the card serves. Conversely, compaction that keeps sessions under a cap is not only a cost and accuracy measure on self-hosted hardware; it is the mechanism that doubles your fleet's capacity without buying a card.

The five numbers to write down before you commit to hardware.

You can do this in an afternoon, and it is the difference between a capacity plan and a hope.

• Per-token cache cost, from the config. Read num_hidden_layers, num_key_value_heads and head_dim, count how many layers are actually attention layers if the backbone is hybrid, and compute the product. Quote it in KB per token and put it in your runbook.
• Weight footprint at your serving precision, measured from the loaded process rather than inferred from the parameter count.
• The context cap you will enforce, as a configured limit with a defined behaviour when a session hits it — compaction, summarisation, or a clean refusal. "We did not set one" means your cap is the context window, and your capacity plan is sized for it.
• The p99 session length you actually see, from traces. If p99 is near the cap, the cap is doing work; if it is a tenth of the cap, you are reserving memory for sessions that never happen and should lower the cap and raise concurrency.
• Measured sessions per card under load, not the computed number. Fragmentation, prefix sharing and offload all make reality differ from the division, usually in both directions. Load testing at the cap is how you find out which.

Related: on-device agent architecture for the rest of the local stack including cache warmth across restarts, self-hosted inference for agents for the operational commitment, small and local models for which jobs never needed the big model, and cost, quality and latency for the hosted side of the same trade.