Batch & asynchronous inference.
Some large share of your model bill is paying interactive prices for work nobody is waiting on. The major providers sell the same models at roughly half the standard input and output rate in exchange for a completion window measured in hours — and the jobs that qualify, evaluation runs and backfills and index builds, are usually the highest-volume things you run. The catch worth knowing early: an agent loop can never use it.
What the trade actually is.
You submit a file of many independent requests, the provider works through them on its own schedule, and you collect results when the job finishes. Anthropic's Message Batches API and OpenAI's Batch API both price this at 50% of standard rates on both input and output, with a 24-hour completion window as the guarantee; in practice most batches land well inside it.
- It is the same model. Same weights, same quality, same context window. You are not downgrading capability — this is the one cost lever that costs you nothing but time.
- The discount has a reason. Schedulable work lets the provider fill troughs in demand between interactive traffic peaks. You are selling latency flexibility, which is why the price is structural rather than promotional.
- Requests are independent. The unit of submission is a list, not a conversation. Each entry carries its own full prompt and gets its own result, keyed by an ID you supply.
Half price on work with no human waiting is not one option among several to be weighed. Compare it with the alternatives you would normally reach for first: switching to a weaker model costs quality, prompt compression costs engineering time and accuracy, and neither reliably delivers 50%.
Which of your work is secretly offline.
Apply one test: is a person blocked on this in the next few seconds? If not, it is a batch candidate. In most systems the list is longer than expected.
- Evaluation runs. A fixed set of inputs scored on a schedule — the purest batch shape there is, and often the single largest non-production consumer of tokens. See evaluating agents.
- Backfill and enrichment. Classifying, summarising, or tagging a table you already have. The work is bounded, the deadline is "before the next report."
- Index construction. Document summaries, extracted entities, and generated queries that feed a retrieval system are built once and read many times — see chunking and vector search.
- Data generation. Producing training or test examples at volume, where a day of turnaround changes nothing; see synthetic data.
- Scheduled reporting. The nightly digest that runs at 03:00 and is read at 09:00 has six hours of slack it is currently not spending.
Where batch does not fit — the important half.
The limits are structural, and knowing them stops you from designing a system that cannot use the discount.
- A loop cannot be batched. Step n+1 of an agent loop takes step n's output as input, so the requests do not exist yet when the batch is submitted. Batch fits fan-out shapes — many independent calls — not sequential ones. You can batch a thousand agent tasks; you cannot batch one task's steps.
- It pulls against prompt caching. Cache entries expire on the order of minutes while a batch is dispatched over hours, so a shared prefix is unlikely to stay warm across the job. If most of your saving already comes from a large cached prefix, measure both paths rather than assuming they stack — see prompt caching.
- No steering mid-flight. There is no partial output to inspect and no way to intervene once submitted. Anything that needs a human in the middle belongs on the interactive path.
- The window is a ceiling, not a promise. Most batches finish fast; you cannot plan a product feature on that. If a deadline is hard, you need a synchronous fallback for whatever is still outstanding.
Build two lanes, routed by deadline.
The design that captures this cleanly is not a batch feature bolted onto a request path. It is a queue where every unit of work carries a deadline, and the deadline — not the model, not the task type — picks the lane.
- Deadline over one hour → batch lane. Everything else goes synchronous. This is a routing rule you can state in one line, which is what makes it survive contact with a growing codebase. It composes with, but is independent from, model routing: one decides which model, the other decides how urgently.
- Degrade to sync on the tail. Submit to batch, and if the deadline arrives with results outstanding, re-issue only those requests synchronously. You pay full price on a handful instead of on everything.
- Make the job resumable. Track submitted, returned, and failed per request ID. Individual entries fail individually, and a batch job that cannot be partially retried will be re-run whole — at which point you have spent the discount twice.
- Put the split on a dashboard. Percentage of monthly tokens flowing through the batch lane is a number most teams have never computed. It usually starts near zero and has a great deal of room.
Start with your eval runs: pure batch shape, no one waiting, frequently the biggest token line item outside production, and moving them is a configuration change rather than an architecture change. Then take the nightly enrichment job. Only after those two are converted is it worth restructuring a user-facing feature around an async lane — that is real product work, and you should have banked the free savings before you spend the engineering.
Related: agent cost control for the levers in order of return, cost, quality & latency for the underlying triangle, and serving and access for where these APIs sit.