Agentic AI Wiki
AI Blog / vLLM vs SGLang vs TensorRT-LLM vs llama.cpp: Throughput Is the Wrong Benchmark for Agents
AI Blog

vLLM vs SGLang vs TensorRT-LLM vs llama.cpp: Throughput Is the Wrong Benchmark for Agents

Every comparison of these four opens with tokens per second on a fixed batch — the one number that transfers worst to agent traffic, where the same prompt comes back twenty times with a few hundred tokens appended. What separates them is what the KV cache is keyed on, whether constrained decoding survives a full batch, and how much of your quarter the build step eats.

By Agentic AI Wiki 13 min read

Pick your serving engine on a throughput benchmark and you will optimise the one number your agents never exercise. A chat benchmark sends fresh prompts to a fixed batch; an agent sends the same prompt back twenty times with a few hundred tokens appended, so the work is overwhelmingly prefill of text the GPU has already seen. Three things decide whether that work happens twice — what the KV cache is keyed on, whether constrained decoding survives a full batch, and how long the build step takes — and only one of the four engines below was designed with all three in mind.

At a glance

Four projects that all serve open-weight models over an HTTP endpoint, and share almost nothing else about how they get there.

ProjectSinceLicenceDeployment shape
vLLM Feb 2023 Apache-2.0 Python server, one process per GPU group; the default fleet choice.
SGLang Jan 2024 Apache-2.0 Python server, same shape as vLLM; built around a prefix-sharing cache.
TensorRT-LLM Aug 2023 NVIDIA terms Compile an engine for a specific GPU and precision, then serve it.
llama.cpp Mar 2023 MIT One C++ binary and one GGUF file; laptop to single server.
GitHub stars — llama.cpp, vLLM, SGLang, TensorRT-LLM Horizontal bar chart of GitHub stars in thousands, snapshot 1 August 2026: llama.cpp leads at 122.3k, vLLM 87.8k, SGLang 31.0k, TensorRT-LLM 14.3k. Stars track how many people run a project on their own machine, so the ordering here is close to the inverse of datacentre deployment share. GitHub stars (thousands, 1 Aug 2026) 0 25k 50k 75k 100k 125k stars llama.cpp MIT, since Mar 2023 122.3k vLLM Apache-2.0, since Feb 2023 87.8k SGLang Apache-2.0, since Jan 2024 31.0k TensorRT-LLM NVIDIA terms, since Aug 2023 14.3k
Stars measure how many people run something on their own machine, which is why this ordering is close to the inverse of datacentre deployment share.

Both Python servers ship at a pace that makes any version-specific claim stale within a month — vLLM cut v0.26.0 on 27 July 2026 with 411 commits from 212 contributors, SGLang cut v0.5.16 two days earlier with 574 pull requests from 169. Treat feature-parity gaps as temporary and architectural commitments as permanent, because only the second kind survives a release cycle.

Capability matrix across the four serving engines A four-by-five grid scoring vLLM, SGLang, TensorRT-LLM and llama.cpp on prefix reuse, constrained decoding under a full batch, fleet operations, non-NVIDIA silicon, and time from zero to a serving endpoint. Each engine is strong where its design premise points and against the grain everywhere else. Where each engine leans hardest Prefix reuse Constrained decode at batch Fleet operations Non-NVIDIA silicon Zero to serving vLLM Automatic, paged blocks XGrammar default Deepest router and K8s stack ROCm, XPU, TPU pip, then minutes SGLang Radix tree, shared prefixes Mask overlapped with decode Strong, smaller ecosystem AMD, Xeon, TPU, NPU pip, then minutes TensorRT-LLM Block reuse, engine-bound Available, less exercised Strong, one vendor path NVIDIA only Build per model, GPU, precision llama.cpp Per-slot, one machine GBNF, thin batching Not the target CPU, Metal, Vulkan, ROCm One binary, one file Strong — this is what the design is for Workable Against the grain
Each engine is strong exactly where its founding premise points, and against the grain everywhere else.

vLLM — the default, and mostly deservedly

PagedAttention, and what it bought

vLLM's original contribution was to stop treating the KV cache as one contiguous allocation per sequence. It splits the cache into fixed-size blocks and manages them the way an operating system manages virtual memory pages, which drops fragmentation close to zero and lets a batch pack tightly. That is why vLLM's throughput advantage over naive serving was large enough in 2023 to end the argument, and why every engine here now does some version of the same thing.

The ecosystem is the actual moat

Three years of being the default produced the thing that is genuinely hard to replicate: Kubernetes operators, autoscalers, prefix-aware routers, observability integrations, and a model-support surface that usually has a new architecture working within days of release. It also runs on more silicon than anything else in its class — CUDA, ROCm, Intel XPU and TPU — which matters more than it looks when your capacity plan depends on whatever GPUs you can actually get.

Where it is weakest

Structured output. vLLM defaults to XGrammar for guided decoding, and the constraint mask is computed on the critical path; third-party measurements through 2026 have repeatedly shown throughput degrading once guided decoding is on and the batch grows past single digits. For a chatbot that is a footnote. For an agent, where every tool call is a constrained generation, it is the common case.

SGLang — the one built for the shape agents actually make

RadixAttention: the cache knows about prefixes

SGLang stores the KV cache in a radix tree keyed on the token prefix rather than in flat hashed blocks. The consequence is not a marginally better hit rate — it is a different capability. Two requests that share their first forty thousand tokens share that computation automatically, without anyone declaring a session, tagging a cache key, or routing them together. For a fleet of agents running the same system prompt, the same tool definitions and the same retrieved corpus, that describes essentially all of the input.

Constrained decoding that does not stall the batch

SGLang overlaps grammar-mask computation with the GPU step instead of blocking on it, which is the direct answer to vLLM's weakest axis. If your workload is mostly tool calls and JSON, this is the difference that shows up in your latency percentiles rather than in a benchmark table.

The cost of being second

A third of vLLM's stars, a smaller operator and tooling ecosystem, and fewer people who have hit your bug before. Third-party H100 benchmarks in 2026 put SGLang roughly 15–30% ahead of vLLM on shared-prefix workloads; treat that band as directional, since it moves with the model, the batch and which fortnight the benchmark was run.

TensorRT-LLM — fastest, if you can pay the build

Compilation is the whole trade

TensorRT-LLM does not interpret a model graph at runtime. It compiles the model into an engine tuned to one GPU architecture and one precision, which is how it posts the fastest absolute numbers on NVIDIA hardware. The price is a build step measured in tens of minutes per combination of model, GPU and precision, and a rebuild whenever any of those change.

What that does to your operations

Every deployment property downstream inherits the compile step. Mixed fleets need an engine per GPU type. Trying a new quantisation is a build, not a flag. Rolling back a model version means having kept the old engine artifact. If you run one model on one homogeneous NVIDIA fleet at high volume, that is a fixed cost you amortise happily; if you are still moving between models every few weeks, it is a tax on exactly the thing you do most.

NVIDIA-only, by design

This is not an oversight to be fixed. The performance comes from compiling to specific NVIDIA kernels, so hardware portability is the thing being traded away. Take it as a deliberate bet on a vendor, and price the lock-in on purpose rather than discovering it during a capacity crunch.

llama.cpp — the one that is not competing

A different problem, solved unusually well

llama.cpp is the most-starred project here by a wide margin and the least relevant to a serving fleet, because it was built to run a model on one machine you own — CPU, Apple Silicon, Vulkan, ROCm or CUDA — from a single quantised GGUF file with no Python and no CUDA toolchain. That is why it has 122,000 stars and why it appears in almost no production agent fleets.

Where it wins outright

Anything on the developer's machine, anything in an air-gapped or edge deployment, anything where the model must ship inside the product. It also has the shortest distance from zero to a serving endpoint of anything in this comparison: one binary, one file, an OpenAI-compatible server, and GBNF grammars for constrained output.

Where it stops

Multi-tenant concurrency. The server keeps prompt caches per slot and can reuse a common prefix, but it is designed around a handful of slots on one box rather than hundreds of concurrent sequences across a fleet. Running an agent workload on it is not wrong — it is a category error that shows up as a queue.

The three axes that actually decide it

1. Prefix reuse — and it is a routing problem, not an engine one

What each engine keys its KV cache on Four columns comparing cache structure: vLLM hashes fixed-size paged blocks, SGLang keeps a radix tree over token prefixes so overlapping conversations share automatically, TensorRT-LLM reuses blocks inside a compiled engine, and llama.cpp keeps a per-slot prompt cache on a single machine. What the KV cache is keyed on vLLM Fixed-size blocks, hashed by content Cross-request reuse, routing left to you SGLang Radix tree over the token prefix Overlapping runs share automatically TensorRT-LLM Paged blocks inside a compiled engine Reuse is bound to the shape you built llama.cpp Prompt cache per server slot One process, a handful of slots All four can skip a repeated prefill. Only one of them decides, on its own, that two different conversations share the first forty thousand tokens.
All four can skip a repeated prefill. They differ in whether the cache can notice that two conversations overlap.

SGLang's radix tree makes overlap automatic, vLLM's content-hashed blocks make it available, TensorRT-LLM's reuse is bound to the engine you built, and llama.cpp's is bound to one process. But that ranking evaporates the moment you put more than one replica behind a load balancer, because none of them can reuse a cache they were never sent.

Prefix-cache reuse is decided by the router, not the engine Step 20 of an agent loop re-sends its whole transcript. A prefix-aware router sends it to the replica already holding that conversation's KV cache, so only the 300 new tokens are prefilled. A round-robin router sends it to a cold replica, which re-prefills all 60,000 tokens — the same engine, two orders of magnitude apart in work done. Agent loop, step 20 60,000 tokens of prefix + 300 new Prefix-aware router pin session to its cache Round-robin router next replica in the ring Replica A — cache warm holds steps 1–19 prefill 300 tokens Replica C — cache cold holds nothing for this run prefill 60,300 tokens Same engine. Same weights. Same GPU. 200× the prefill work.
Same engine, same weights, same GPU. The router decides whether you do 300 tokens of work or 60,300.

This is the finding that should reorder most migration plans. Teams routinely spend a quarter moving between engines to chase a 20% throughput difference while running round-robin balancing that discards a large fraction of their prefix cache — a difference measured in multiples, fixed by a routing config. Measure your own prefix-cache hit rate before you compare anyone's benchmark, and if it is below about 80% on agent traffic, the engine is not your problem.

2. Constrained decoding, at the batch size you actually run

Every engine here supports grammar-constrained output; the differences appear only under concurrency. SGLang overlaps mask computation with the GPU step, vLLM computes it on the critical path with XGrammar as the default backend, TensorRT-LLM supports it but has it less exercised in the open, and llama.cpp's GBNF is excellent for one stream and not the point at a hundred. Since an agent emits a constrained generation on every single tool call, this axis is load-bearing for agents in a way it simply is not for chat — and it is the axis most comparison posts omit entirely, because their benchmark ran unconstrained.

3. What the setup step costs you, repeatedly

llama.cpp gets you serving in a minute, the two Python servers in an afternoon of dependency work, and TensorRT-LLM after a compile per model-GPU-precision triple. The relevant question is not the one-time cost but the recurring one: how often do you change model, quantisation or hardware? A team pinning one model for a year barely notices TensorRT-LLM's build. A team evaluating a new open-weight release every few weeks pays it every time, and that recurring cost usually swamps the throughput advantage it bought.

When to pick which

SituationPickBecause
General agent fleet, mixed models, a team that has to operate itvLLMThe ecosystem, the hardware breadth, and the largest supply of people who have debugged it.
High-concurrency agents on a stable prompt and toolsetSGLangAutomatic prefix sharing and constrained decoding that does not stall the batch — the two things agent traffic hammers.
One model, homogeneous NVIDIA fleet, latency is the productTensorRT-LLMFastest absolute numbers, and the build cost amortises when nothing changes.
Local, edge, air-gapped, or shipped inside the appllama.cppOne binary, one file, no toolchain, runs on the hardware you already have.
You are not sure yetvLLM, with prefix-aware routing onGet the routing right first; the engine swap stays cheap and the routing win is larger anyway.

FAQ

Is SGLang faster than vLLM?

On shared-prefix workloads, third-party H100 benchmarks in 2026 have put it roughly 15–30% ahead, and its architecture is the reason. But that gap is smaller than the one you create by load-balancing agent traffic round-robin, so fix routing before you migrate.

Why is llama.cpp the most-starred if it is the least deployed?

Stars measure people who ran something themselves, and llama.cpp is what most people use to run a model on a laptop. Datacentre serving is a much smaller population making a different choice.

Does TensorRT-LLM's compile step really take that long?

Tens of minutes per model, GPU architecture and precision combination is the number to plan against. It is a fixed cost, not a per-request one — the question is how often your configuration changes.

Can I run these on AMD or Apple hardware?

vLLM runs on ROCm, Intel XPU and TPU; SGLang reports AMD, Intel Xeon, TPU and NPU support; llama.cpp runs on nearly anything including Apple Silicon via Metal. TensorRT-LLM is NVIDIA-only by design.

What should I measure before choosing?

Prefix-cache hit rate on your real traffic, time-to-first-token split by cache hit and miss, and end-to-end task completion time rather than tokens per second. If those three are not instrumented, no benchmark comparison will predict your outcome.

Further reading

On this wiki:

Project sources: