Load-Testing an Agent System

Nothing you own saturates first.

A web service runs out of CPU. An agent system runs out of somebody else's quota, and it does so at traffic levels that leave your own infrastructure idle. Capacity here is concurrent in-flight tasks, not requests per second, and the ceiling is almost always one of four things — none of which a conventional load test is pointed at:

• Provider tokens-per-minute. Agent token burn is quadratic in steps, because every step re-sends the transcript, while your quota is flat. This lands long before request-per-minute limits do; the arithmetic is in rate limits and provider capacity.
• A third-party tool's rate limit. Usually the least generous number in the system and the one nobody inventoried. One agent task can make thirty calls to the same CRM.
• Worker slots pinned by the tail. Long-tailed task durations mean the p99 run holds a worker for minutes while short runs queue behind it.
• KV-cache memory, if you self-host: capacity is concurrent sequences times context length, not a request count — see self-hosted inference for agents.

Size the test from Little's Law before you run anything: concurrency equals arrival rate times mean task duration. Ten tasks a minute at a four-minute mean is forty in flight — and if your p99 is twenty minutes, the tail alone is holding a meaningful share of your workers at all times. That number tells you what load to generate; anything below it is a smoke test wearing a load test's name.

Decide the side-effect question first, and know what it costs you.

Agents call tools that book, post, delete and charge. You cannot run a thousand of them without answering this, and each answer buys a different measurement. Pick deliberately rather than defaulting to whichever your test framework makes easy.

• Mocked tools. Safe, cheap, repeatable — and systematically wrong about the thing you are testing. A real tool call takes hundreds of milliseconds to seconds; a mock takes microseconds. Trajectories finish far too fast, concurrency never builds, and you conclude you have capacity you do not have. If you mock, replay a recorded latency distribution per tool, tail included, not a fixed sleep — the tail is what pins workers.
• Recorded replay. Real latencies and real payload sizes, no side effects. The catch is that the model goes off-script: it will call something the recording does not contain. Make a cache miss a counted failure, never a synthesised response — a fabricated tool result quietly changes the trajectory and invalidates the run.
• Real tools against a sandbox tenant. The only option that measures downstream rate limits honestly, which is to say the only one that finds the ceiling that will actually bite. It costs real money and real quota. Tell the vendor before you run it; a sudden 50× in call volume from one account looks exactly like an attack, and being throttled mid-test teaches you nothing you can act on.
• Production, read paths only. Useful for retrieval and context assembly, and strictly bounded: the write tools must be removed from the catalog, not merely discouraged in the prompt.

Generate tasks, not repeated requests.

The standard load-generation instinct — pick one request, fire it ten thousand times — produces a badly misleading result here, because the second identical prompt hits a warm prompt cache and every one after it is cheaper and faster than anything production will ever see. You will measure a cache and call it a system.

• Draw the task mix from production traces. What matters is the distribution of difficulty, and therefore of step counts: a workload that is 90% three-step and 10% forty-step behaves nothing like its mean. Sample real tasks, do not invent a representative one.
• Vary the prefix realistically. Different tenants, different tool subsets, different retrieved documents. Cache hit rate is an input to your capacity, so make it match production rather than accidentally pinning it at 100%.
• Report cold and warm separately. Both are real operating states — a deploy that changes the system prompt invalidates every cache at once, which is precisely when traffic is unchanged and your costs and latency both jump.
• Treat context length as a load axis. A test that runs only short contexts overstates capacity, sometimes by an order of magnitude, because the memory a run occupies scales with its transcript.
• Include the failure paths. Real traffic contains tool errors, timeouts and retries, and retries are where load multiplies. A test with a 0% tool-error rate cannot find retry amplification, which is the mechanism behind most agent-system outages.

Where you need pressure on the conversational surface rather than the task queue, the harness for it already exists as an eval technique — see simulated users in agent eval.

Assert on quality, because degradation is silent.

This is the step teams skip and the reason their load tests pass while the system is failing. Under saturation an agent degrades in ways that produce no errors at all: it falls back to a smaller model, truncates context to fit, shortens the thinking budget, drops a retry it would normally make. Latency stays inside the SLO. The error rate stays at zero. The answers get worse.

So a load test must carry an eval, not just a stopwatch:

• Run a fixed subset of your eval set as part of the load. Same tasks, same graders, run at rest and again under load. The comparison is the finding — and a system whose success rate drops eight points at 70% of nominal capacity has a real capacity of 70%, whatever the latency chart shows.
• Count every degradation event explicitly. Fallback invocations, context truncations, effort-budget reductions, shed requests. If degraded mode is not a named state you can count, you cannot tell a healthy test from a test that spent its second half running on the fallback path.
• Watch step count, not just duration. A model under a shortened effort budget frequently takes more steps to reach the same place, which raises token burn exactly when quota is scarce. This is the feedback loop that turns a shortfall into an outage.

Measure the handful of numbers that mean something.

Conventional load-test dashboards report RPS and p99 response time, and for an agent both are close to meaningless — a "request" that lasts four minutes and makes forty downstream calls is not the unit anything saturates on. Instrument these instead, all of them joinable to a trace:

• Concurrent in-flight tasks, plotted against the ceiling you calculated in step 1.
• Task duration distribution — p50, p95, p99 — and steps per task alongside it, since the two move together and only one of them is a cost driver.
• Tokens per minute against quota, as headroom rather than as a total. Headroom is the number that predicts the incident; see measuring agent latency for the timing side.
• Call rate per downstream dependency, and 429s counted per provider and per tool separately. Aggregated, they hide which ceiling you actually hit — the single most valuable output of the whole exercise.
• Retry amplification — attempts divided by logical operations. Anything above about 1.3 under load means your retry policy is part of the problem, not the mitigation.
• Queue depth and queue age. Depth tells you how much work is waiting; age tells you whether anything is starving, and only age catches the run that has been queued for forty minutes behind the tail.
• Dollars per run. A meaningful agent load test costs real money, and the number is itself a result: it is your marginal cost at that traffic level, arrived at honestly.

Make it recurring, because the ceiling moves.

A load test is usually run once before launch and then never again — which is defensible for a service whose capacity changes only when you deploy. It is wrong here, because an agent's capacity moves when the model, the prompt or the tool catalog changes, and two of those three can change without you shipping anything. A provider swaps in a new snapshot; a vendor rewrites a tool description and step counts drift; your own retrieval starts returning longer documents. Each one moves the ceiling silently.

• Re-run on every change to the behaviour tuple — model, prompt, tools, policy — as part of the same gate that runs your evals. It does not have to be a full-scale test; a fixed short run at fixed concurrency, compared against the last one, catches drift.
• Set admission control from the measured ceiling, not the vendor's documented one. Cap in-flight tasks at roughly 70–80% of where quality started to degrade — the number from step 4, not the number where things errored.
• Contain the test itself. Separate credentials, a hard spend cap, and a kill switch you have verified stops in-flight runs and not just new ones. A load test is the highest-volume runaway agent you will ever deliberately create.
• Keep the traces. The runs are a corpus: the slowest 1% of trajectories from a load test are the best available sample of what your agent does when it is struggling, and they read very differently from the failures you see at rest.

Related: concurrency and scaling for the architecture the test is probing, idempotency and retries for why the write path needs keys before you generate any load at all, cost control in the loop for the caps that keep the test itself bounded, and multi-tenancy for agents for the shared pools one loud tenant can exhaust.