Rate Limits & Provider Capacity

Know which of the two limits you are actually hitting.

Providers meter at least two independent dimensions, and they fail in completely different ways. Requests per minute counts calls. Tokens per minute counts the input and output volume those calls move — often split into separate input and output buckets, and often accounted against your declared maximum output tokens rather than what the model actually generated.

Chat products hit the request limit first. Agents almost never do. A twenty-step agent loop is twenty calls — trivial against a request budget — but each call re-sends a transcript that has been growing the whole time, so the same twenty calls can move millions of tokens. Teams size their capacity against request counts, watch the request meter sit at 8% utilisation, and get throttled anyway.

Agent load is quadratic; your quota is flat.

This is the structural mismatch, and it is why capacity planning that worked for a chat feature fails on the first agent deployment. Because the model is stateless, step n re-sends everything from steps 1 through n−1, so total input tokens for a task grow with the square of the step count — the same arithmetic that drives agent cost control, arriving here as a throughput problem instead of a billing one.

Two consequences follow that nobody plans for:

• Your peak is set by task difficulty, not by traffic. A day when tasks happen to run long produces a token spike with no corresponding rise in user count. Autoscaling on request rate sees nothing.
• Concurrency multiplies the tail, not the mean. Ten concurrent agents at step 3 are cheap; ten at step 25 can be an order of magnitude more expensive in tokens per minute. Since agents start together and drift apart, your worst minute is the one where several long runs overlap late.

Plan capacity in tokens per minute at the p99 of concurrent long runs, not at the mean. And note that prompt caching changes the arithmetic but usually not the metering — cached reads still count against most token budgets even when they are billed at a discount, so verify with your provider rather than assuming your cache hit rate bought headroom.

Move the queue to your side of the wire.

When the provider rejects a request you have already paid the latency of a round trip to learn something you could have known locally: that you are over budget. Worse, the standard exponential-backoff-and-retry loop is a positive feedback loop under sustained overload — every throttled caller retries, aggregate demand rises, more callers are throttled, and a 20% capacity shortfall presents as a total outage.

The shape that survives is a local admission controller: a token-bucket rate limiter, shared across your workers, sized slightly below your actual quota. Requests wait in your queue, where you can see them, prioritise them, and shed them — instead of in a retry storm you cannot observe.

# infra/admission.py — spend a budget you can see, not one you discover
BUCKET = TokenBucket(
    tpm=1_800_000,          # 90% of the 2M/min contract: leave headroom
    rpm=900,
    shared="redis://limits",  # one budget across ALL workers, not per pod
)

async def call_model(req, *, priority):
    cost = req.input_tokens + req.max_output_tokens   # reserve the declared max
    if not await BUCKET.acquire(cost, priority=priority, timeout=30):
        raise Shed("no capacity for this class")      # fail fast, fail visibly
    try:
        return await provider.send(req)
    finally:
        BUCKET.settle(cost, actual=req.usage)         # refund the unused reservation

Two details do the work. The budget is shared, because a per-pod limiter divided across a fleet that scales is not a limit at all. And the reservation uses max_output_tokens, because that is usually what the provider reserves too — which is also why capping max_output_tokens to something realistic is one of the cheapest throughput wins available.

Keep retries for what they are for. A 429 you did not cause locally means the quota moved or another consumer is spending it; retry that once, slowly, with jitter, and treat repeated occurrences as a capacity signal rather than a transient. Genuine 5xx and timeouts are a different class and belong in idempotency and retries.

Shed on purpose, because saturation will choose for you.

Once the queue is yours, you own the interesting decision: when demand exceeds capacity, whose work stops. Left unmanaged, the answer is arbitrary — and arbitrary usually means the interactive request a customer is watching gets queued behind a batch backfill nobody is waiting for.

• Class every call at the source. Interactive, background, retry, and speculative are four different priorities. The class travels with the request; it is not inferred at the limiter.
• Reserve a floor for interactive. A fixed share of the budget that background work can never consume. Without it, one large batch job takes the whole product down and the graphs will show a healthy provider.
• Move what can wait off the live budget entirely. Evals, backfills, summarisation and enrichment usually tolerate hours — see batch and async inference, which typically runs on a separate quota and at a discount.
• Degrade before you drop. A smaller model, a shorter context, or fewer parallel branches delivers a worse answer under load, which almost always beats no answer — the routing decision in model routing, triggered by capacity rather than by task difficulty.
• Shed loudly. A shed request must surface as a distinct, counted outcome. Shedding that presents as a generic error is indistinguishable from a bug, and someone will spend a week debugging your own limiter.

Cross-provider failover changes behavior, not just availability.

The reflex answer to capacity risk is a second provider and automatic failover. It is a real mitigation and it is routinely mis-specified, because teams reason about it as an availability feature when it is a deploy: at the moment of failover, a fleet of agents silently begins running on a model your evals never covered.

What actually differs on the other side, in rough order of how much damage it does:

• Tool calling. Schema dialects, parallel-call support, argument coercion, and how strictly JSON Schema is honoured all vary. An agent whose tool loop was tuned on one provider can start emitting malformed arguments on another — the vendor differences catalogued in JSON Schema subsets per vendor.
• Refusal and safety behavior. The fallback may decline work the primary performs, which reads to your users as a mysterious intermittent failure correlated with nothing they can see.
• Prompt caching. Failing over discards a warm cache and can multiply both cost and latency at exactly the moment you are already under pressure.
• Context limits and formatting. A prompt that fits the primary may not fit the fallback, so failover manifests as truncation errors rather than as a clean degradation.

So treat the fallback path as a shipped configuration: run your eval suite against it on the same schedule as the primary, and send a small continuous share of live traffic through it so the path is warm and observed rather than theoretical. A failover route that has never served production traffic is an untested code path that activates only during an incident, which is the worst possible time to discover it. And pin the fallback to a dated snapshot too — everything in rollout, versioning and pinning applies twice as hard to a model you rarely look at.

Make headroom a number somebody watches.

Capacity is a contract with a lead time. Quota increases are negotiated, sometimes over days, occasionally with a commitment attached — so the useful signal is not "are we being throttled" but "how close are we, and how fast is that closing".

• Graph utilisation per dimension — tokens per minute and requests per minute, as a percentage of contracted quota, per provider and per key. Alert at 70% sustained, not at the first 429. By the time you are being rejected, the negotiation you needed to start is already late.
• Alert on the derivative. A new feature that doubles tokens per task shows up as a slope days before it shows up as an incident.
• Track shed rate by class. Rising background sheds are healthy load management; a single interactive shed is a customer-visible failure and should page differently.
• Know your ceiling under failure. Run the arithmetic for "primary at zero, everything on the fallback" and confirm the fallback's quota can actually absorb it. Most cannot, and discovering that during the failover is how a partial outage becomes a total one.
• Separate keys by workload. Distinct keys or projects for interactive, batch, and evals give you per-workload meters and stop a runaway eval job from consuming the production budget — the containment principle behind kill switches.

Related: concurrency, queues and scaling for the worker pool this limiter sits in, cost control at the loop level for the same arithmetic priced in dollars, and inference providers for how quota differs across the market.