Forecasting Agent Spend

Why the average cannot forecast an agent's bill.

A chatbot's cost per request is tightly clustered — one call, a bounded response, a distribution you can summarise with its mean. An agent's cost per task is not. Cost grows with the square of step count (every step re-sends the whole transcript, the arithmetic worked in agent-cost-control), and step count itself has a long right tail: most tasks finish in a few steps, a few loop, retry, or wander for dozens. Multiply a quadratic by a long-tailed input and you get a distribution where the top 5% of runs can be a third of the spend.

Two things follow, and both break the naive forecast:

• The mean is not a typical run. It sits above the median, pulled up by the tail, so "average cost per task" describes no task anyone actually ran. Reporting it invites everyone to reason about a run that does not exist.
• Mean × volume is biased low, and gets worse with scale. A larger population draws more samples from the tail, so the realised average per task creeps up as volume grows. The forecast that undershot at ten thousand tasks undershoots by more at a million.

Forecast a task mix, not a blended average.

The fleet-wide distribution is a sum of narrower ones, and the sum hides everything actionable. A support-triage task, a code-migration task and a research task have different cost shapes; blending them into one average forecasts none of them and misleads when the mix shifts. Build the forecast as a sum over task classes:

• Segment by task class, not by model or endpoint. The unit of prediction is the kind of work, because that is what has a stable cost distribution and what product changes when it ships a feature.
• Carry each class's distribution, not its mean. Keep the p50, p95 and p99 per class — or better, resample the class's own historical costs — and sum across classes to get the fleet number with its spread intact.
• Treat the mix as the volatile input. A forecast pinned to today's mix is stale the moment a new agent ships. When product plans a launch, the forecast input that changes is the class weights, and the honest forecast updates before the launch, not after the invoice.

This is the same per-segment discipline that per-customer-economics applies to tenants, turned toward task types: an aggregate that looks stable can be two classes moving in opposite directions.

The cap is what makes the tail forecastable.

A heavy-tailed distribution with no ceiling has no dependable expectation to budget against — the more runs you observe, the higher the sample mean climbs, because the next extreme is always ahead of you. The per-task ceiling from cost-control-in-the-loop is not only a safety control; it is what converts an unbounded risk into a line item. With a hard cap at C tokens (or dollars, or steps) per task, the worst run costs C, the tail is truncated, and the fleet cost has a finite, computable upper bound.

So forecast the tail explicitly rather than hoping it stays small:

• Track the cap-hit rate per class. The fraction of runs that reach the ceiling is the mass of the truncated tail, and it is a leading indicator: a class whose cap-hit rate is rising is a class whose cost is about to.
• Budget a worst case, not just an expected case. Expected spend uses each class's mean; the commitment you make to finance uses a high percentile so a bad month lands inside the number you promised. The gap between them is the tail's price, and naming it is the point.
• A missing cap is a missing forecast. If any task class runs uncapped, its contribution to the total is genuinely unbounded and no forecast over it is honest. Cap first, then predict.

The four drivers that move the tail.

A spend forecast is a function of a small number of inputs, and the tail is far more sensitive to them than the median is. Instrument these so the forecast is fed by live numbers rather than a quarterly guess:

• Retry and loop rate. The single largest tail driver. A failed task that retries pays for the failure and the retry, and a retry storm during a provider wobble (see rate-limits-and-provider-capacity) can double a day's spend while producing nothing.
• Context growth. The quadratic lives here. A class that started returning large tool results into the transcript will see its p95 climb long before its median moves — watch the tail, not the average, to catch it.
• Route mix. Which steps go to the frontier model versus a small one. A cascade that silently shifts more traffic to the expensive model reprices the whole class; this is also where a model migration can move the base without anyone editing a forecast.
• Cache hit rate. Prompt-cache hit rate multiplies the effective input price. A deploy that reorders the prompt prefix and tanks the hit rate raises every run's cost at once — a step change the forecast must ingest, not a gradual drift.

From forecast to a budget that acts, then reconcile.

A forecast that only lives in a slide is a prediction; a forecast wired to controls is a budget. Use three numbers from the same distribution for three different jobs:

• p50 for the plan. The expected-case number finance builds the quarter around.
• p95 for the commitment and the alert. The number you promise not to exceed, and the threshold that pages when a month is tracking to blow through it — early enough to act, per cost-attribution's rule that budgets are circuit breakers, not reports.
• The spend rate for the circuit breaker. Tokens-per-minute crossing a slope, not dollars-per-month crossing a total — the derivative arrives while you can still stop the run.

Then close the loop every month: put forecast next to actual and decompose the miss into three terms — volume (more tasks than planned), mix (the class weights moved), and per-task drift (a class got more expensive at constant volume). A miss you cannot decompose teaches you nothing; a miss you can tells you exactly which input to re-estimate. Most persistent under-forecasts turn out to be per-task drift hiding in the tail — the class that quietly started looping.

Where the forecast still breaks.

Even a tail-aware, mix-based forecast has blind spots worth naming so they surprise you less:

• A new task class with no history. Cold-start has no distribution to sample. Bound it with a synthetic worst case — assume it hits the cap far more often than a mature class until real runs say otherwise — rather than borrowing another class's mean and calling it a forecast.
• Silent repricing under you. A provider price change or a platform auto-upgrade reprices the base with no commit in your repo; a term commitment (see provisioned-throughput-and-commitments) freezes price but also your model choice. Stamp the price schedule the forecast assumed, so a base change is a visible input rather than an unexplained miss.
• Cost that correlates with value. Your most engaged customers run the deepest loops, so demand and per-task cost move together and the tail is not evenly distributed across tenants. A fleet forecast that ignores the correlation will misprice exactly the accounts that matter most — the failure surface catalogued in economics-failure-modes.