Agent cost control.
A chatbot's bill scales with users; an agent's bill scales with users times steps times context, and that third factor is the one that surprises people — because on every turn the whole conversation is re-sent, so a twenty-step task does not cost twenty messages, it costs something closer to the square. The fix is almost never a cheaper model. It is caching what repeats, capping what runs away, and deleting steps that were never needed.
Why the arithmetic is quadratic.
Models are stateless. Each step of an agent loop re-sends the system prompt, the tool definitions, and the entire transcript so far — including every tool result already returned. Step 1 sends the prompt; step 20 sends the prompt plus nineteen rounds of accumulated output.
- Input tokens grow roughly linearly with step count, so total input across a task grows with the square of it. Doubling the number of steps roughly quadruples input spend.
- Output tokens grow linearly and are typically 3–5× the price of input per token — but on a long agentic task input volume dwarfs output, so input usually dominates the bill anyway.
- A single fat tool result poisons every subsequent step. Dumping a 50,000-token file into the transcript at step 3 means you pay for those 50,000 tokens again at steps 4 through 20.
This is why "just use the cheap model" disappoints. A model at a fifth of the price that needs three times the steps to finish, and re-sends a transcript that is growing the whole time, can easily cost more — and it will certainly be slower. Capability that ends the task sooner is a cost lever, not a luxury.
The four levers, in order of return.
- Prompt caching. The single largest win available and the least invasive. The stable prefix — system prompt, tool definitions, retrieved documents — is cached and re-read at a large discount instead of reprocessed. It pays out precisely because agents re-send the same prefix on every step. It also requires that the prefix actually be stable, so put anything that varies at the end. See prompt caching.
- Context discipline. Do not put in what you will pay to carry. Summarise or drop old turns, return tool results by reference ("wrote 300 lines to
report.md") rather than by value, and truncate at the tool boundary rather than hoping the model ignores the noise. This is the whole subject of context engineering, and cost is one of its two payoffs — the other being accuracy. - Routing. Send the easy, high-volume steps to a smaller model and keep the frontier model for the reasoning loop. Classification, extraction, and reranking are the usual candidates; see small & local models.
- Doing less. The cheapest token is the one never generated. If a deterministic lookup answers 40% of requests, route those away from the model entirely — and notice that not every task needs an agent.
Caps, because loops do run away.
Optimisation reduces the average. Caps are what stop the tail from becoming an incident, and an agent that can call tools in a loop will eventually loop. Budget at three levels:
- Per task — a hard ceiling on steps and on total tokens. When it trips, stop and hand back what you have; do not silently continue. This is the same control surface as termination.
- Per user or tenant — a rolling quota, so one caller cannot consume the shared budget. Without it, your cost ceiling is set by your most enthusiastic user.
- Global — a kill switch and an alert on spend rate, not just on spend total. A runaway loop shows up as a slope long before it shows up as a monthly invoice.
Alert on the derivative. "Spend is up 12% this month" arrives weeks late; "tokens per minute tripled at 03:14" arrives while you can still do something about it.
Measure per task, not per token.
Token price is the least useful cost metric available, because it tells you nothing about whether the work got done. The number worth putting on a dashboard is cost per successfully completed task — spend divided by tasks that passed your evaluation, not tasks that returned a response.
- It prices failure honestly. A cheap configuration that fails a third of the time is paying full price for those attempts and then paying again for the retry — or paying a human to clean up.
- It makes the model comparison real. A pricier model that finishes in eight steps instead of twenty-five is usually cheaper per completed task, and only this metric shows it.
- It exposes the long tail. Mean cost per task hides the 1% of runs that cost 100×; watch the p95 and p99, which is where a loop that never terminated is hiding.
- Attribute spend to a trace. Without per-run token accounting joined to the run's outcome you are guessing — which is one more thing observability pays for.
Do the three things that need no architecture change first: turn on prompt caching, stop returning whole file contents into the transcript, and set a per-task step ceiling. In most agent systems those three together move the bill more than any model swap, and none of them costs you capability. Only after that is it worth arguing about which model to route where.
Related: cost, quality & latency for the three-way trade-off, context windows for the ceiling all of this runs into, and kill switches for stopping a run in flight.