Self-Hosted Inference for Agents

Capacity is concurrent sequences × context length, and QPS is meaningless.

Every token in flight holds a slice of KV cache for the entire life of the request, and an agent request lives for minutes. The arithmetic is unforgiving and exact:

# KV cache bytes per token (per sequence)
bytes_per_token = 2 * layers * kv_heads * head_dim * dtype_bytes
#          K and V ---^

# 8B-class GQA model: 32 layers, 8 KV heads, head_dim 128, fp16
#   2 * 32 * 8 * 128 * 2  =  131,072 B  =  128 KiB / token
#   128k-token context    =  16 GiB for ONE sequence

# 80 GB card, 16 GB of weights, ~60 GB left for cache:
#   60 / 16  ~=  3.7 concurrent agents at full context

Read that last line twice, because it is the number that surprises teams. The GPU is not compute-starved and the model is not large; it is out of room to remember four conversations. The consequences follow directly:

• Plan in sequence-slots, not requests per second. A dashboard of QPS tells you nothing about whether the next agent can start. The number to alert on is cache occupancy and the count of sequences waiting for a slot.
• Long context is a capacity cost, not just a bill. Cutting a run from 128k to 32k of working context does not shave 75% off a token invoice you no longer receive — it quadruples the number of agents the same card can hold. Context engineering stops being an economy measure and becomes a scaling one.
• Quantise the cache before you buy hardware. An fp8 KV cache halves bytes per token and roughly doubles concurrency, at a quality cost you can measure in an afternoon on your own eval set. It is the highest-return knob on this list and the one most often left at the default.
• Preemption is the pressure valve, and it is not free. When cache runs out, engines evict a sequence and recompute it later. Under load you can spend a large fraction of your compute re-prefilling work you already did — which shows up as latency that gets worse with load in a way throughput graphs hide.

Prefix-cache hit rate is the lever, and it is a routing problem.

An agent loop re-sends its whole transcript every step, so by step twenty the prompt is mostly identical to step nineteen's. Every serving engine can reuse the cached prefix and skip that prefill — which turns the dominant cost of agentic serving into almost nothing. Then a load balancer sends step twenty to a different replica, the cache is cold there, and you re-prefill the entire conversation from scratch.

This is the difference between a fleet that serves agents well and one that does not, and it has nothing to do with which engine you picked:

• Route by prefix, or at minimum by session. Pin a conversation to the replica that already holds its cache. Round-robin is actively wrong for agent traffic in a way it is not for chat.
• Instrument the hit rate as a first-class SLI. Prefix cache hit rate per replica, plus time-to-first-token split by hit and miss. A fleet drifting from 90% to 60% hit rate loses far more capacity than any engine upgrade returns, and nothing else on your dashboard moves.
• Keep the prefix stable and put the variable part last. The same rule as prompt caching on a hosted API — a timestamp near the top of the system prompt invalidates everything after it, on every request.
• Beware the deploy. Rolling a new system prompt or tool schema invalidates every cache in the fleet at once. Expect a prefill storm on release and stage it rather than discovering it.

Two workloads with opposite shapes, sharing one GPU.

Prefill is compute-bound and bursty; decode is memory-bandwidth-bound and steady (see prefill & decode). Agents make the split pathological: a long tool result arrives and triggers a huge prefill in the middle of a fleet of sequences that were quietly decoding, and every one of their inter-token latencies spikes. On a shared endpoint, one agent pasting a 50,000-token file stalls everyone else's stream.

• Cap prompt length at admission, not at the model. A per-request token ceiling enforced before scheduling is the only thing that stops one oversized prompt from becoming everyone's latency incident.
• Separate the lanes when the latencies differ. Interactive agents and background batch work on the same replicas means the batch job's prefills set the interactive tail. Split the fleet or split the priority class; do not average them.
• Chunk long prefills. Every serious engine can interleave a large prefill with ongoing decode in pieces. It is usually a flag, it is usually off by default in older configs, and it is the difference between a hiccup and a stall.
• Watch the right latency. For an agent, end-to-end task time is what matters, and it is dominated by step count × per-step latency. Optimising tokens-per-second while step count grows is optimising the wrong factor — the same trap as cost per token versus cost per task.

The ops surface you just inherited.

The provider was absorbing more than capacity. Self-hosting hands you a stateful, slow-starting, expensive-to-idle service, and the operational assumptions from stateless web services do not transfer.

• Autoscaling does not work at agent timescales. Cold start means allocating a GPU, pulling tens of gigabytes of weights, and loading them — minutes, not seconds, and that is when capacity is available at all. You provision for peak, you queue, or you burst to an API. There is no fourth option, and "we'll scale up when traffic arrives" is the plan that fails during the incident.
• Fairness must be enforced in tokens. One tenant running 200k-context agents will consume the cache while a hundred short requests wait. Per-tenant limits denominated in sequence-slots and tokens, not request counts — the same admission discipline as provider capacity, except now you are the provider and the 429 is yours to issue.
• You own the upgrade risk in both directions. Nobody deprecates your model for you, which removes the forced migration of retirement — and equally means nothing stops you running a stale model for two years. Meanwhile engine upgrades change sampling behaviour, kernel numerics and default flags; pin the engine version alongside the weights and re-qualify on your eval set, because "same weights" does not mean same outputs.
• Keep a warm fallback. A single-region GPU fleet with no API path is a single point of failure with a multi-minute recovery. The cheapest insurance is a provider key and a router that can fail over mid-incident.

The economics turn on utilisation, and agent traffic is bursty.

An on-demand H100 runs roughly $2–3 an hour on specialist GPU clouds in 2026 and appreciably more on hyperscalers — call it $1,500–2,200 a month per card, paid identically whether it is saturated or idle at 4am. That is the whole equation: you converted a variable cost into a fixed one, so the break-even is a utilisation number, and utilisation is exactly what a diurnal, bursty agent workload is bad at.

• Compute the honest break-even. Monthly GPU cost divided by tokens actually served at your real utilisation — not at benchmark throughput. A fleet sized for peak and idle two-thirds of the day is paying triple its headline per-token rate.
• Fill the trough deliberately. Batch work — evals, embeddings, offline classification, synthetic data — is what makes the arithmetic work, and it is why self-hosting suits teams that already have background inference to run.
• Price the people. The GPU bill is the visible half; the engineer maintaining engine versions, cache routing and capacity is the other, and it does not scale down when traffic does.
• Self-host the high-volume small stuff first. Embedding, reranking, routing and extraction are steady, cheap and fit a small model — the argument in small & local models — while the frontier reasoning loop stays on an API. Hybrid is usually the right answer, and it is rarely the one debated.

When self-hosting is right anyway.

Three cases survive the arithmetic even when the per-token comparison does not: data that legally cannot leave your boundary (residency and sovereignty), a fine-tuned model that no provider serves, and sustained high-volume traffic on a small model where utilisation is genuinely high. Everything else is usually a hybrid. Whichever you pick, instrument cache occupancy and prefix-cache hit rate on day one and plan capacity in concurrent sequences at your real context length — a fleet sized on tokens per second will look correctly provisioned right up to the moment it cannot admit a fifth agent.

Related: concurrency, queues & scaling for the job-queue layer above this, serving & access for where self-hosting sits among the options, and vLLM vs SGLang vs TensorRT-LLM vs llama.cpp for the engine comparison itself.