A chat product needs several hundred concurrent users before a dedicated GPU beats per-token pricing. An agent needs about a dozen workers, because it re-sends its whole context on every step and burns tokens an order of magnitude faster per unit of wall-clock time. That arithmetic — not price per million tokens — is what should decide which of these four you build on, because the one thing you will need from them is a way off the rung you started on.
At a glance
Four hosted-inference providers an agent team will shortlist in 2026, and the bet each one made.
| Provider | Core bet | Entry pricing | Dedicated capacity |
|---|---|---|---|
| Together AI | Broadest open-model catalog, plus tuning and raw compute under one contract | Per token — around $1.04/M on the 70B class | Dedicated endpoints, H100 near $6.49/hr; HGX clusters from about $3.99/hr per GPU |
| Fireworks AI | Speed-obsessed serving on a custom engine, priced under the field | Per token — around $0.90/M on the 70B class | On-demand deployments, H100/H200 near $7/hr |
| Baseten | Your weights in production, on capacity brokered across many clouds | Per token on a curated model shortlist | Dedicated deployments, H100 near $6.50/hr |
| Modal | Python-native containers with very fast cold starts; you bring the serving stack | No per-token rung — GPU seconds from the first request | Per-second billing with scale to zero |
Agents cross the break-even where chat products never do
Start with the only arithmetic that matters, and do it once. A dedicated H100 at roughly $7.00 an hour, set against per-token serving at roughly $0.90 per million on a 70B-class open model, buys about 7.8 million tokens of serverless spend per GPU-hour. That is the line. Everything below it is cheaper on tokens; everything above it is cheaper on hardware.
A chat user generates perhaps twenty thousand tokens in an hour of genuinely heavy use. You need close to four hundred of them, concurrently, all hour, to hit the line — which is why serverless per-token pricing has been the obviously correct default for everything conversational, and why the whole industry priced itself around that assumption.
An agent breaks the assumption structurally rather than by being popular. A single trajectory re-sends the system prompt, the tool definitions, the accumulated observations and the prior reasoning on every step, so a twenty-step task with a thirty-thousand-token working context consumes something like 650,000 prompt tokens against maybe eight thousand tokens of output. One agent worker therefore burns roughly what thirty chat users burn, and a dozen of them running continuously spend an H100-hour every hour. Twelve concurrent workers is not a scale milestone. It is a Tuesday for a team running background coding agents.
Crossing the break-even in spend is not the same as being able to serve the load on one GPU. A single H100 running a 70B model may or may not sustain twelve concurrent multi-step trajectories at the latency you need — that depends on quantisation, batch shape and how much of each context is cached. The arithmetic tells you when to run the experiment, not what its result will be.
The consequence for provider choice is direct. If your workload is agentic, you are not picking a per-token vendor for the next three years. You are picking the vendor you will want to still be with on the day per-token pricing stops making sense — which is a question about the ladder, not the entry rung.
Prefix caching is where the agent bill is actually won
The re-sent context that makes agents expensive is also the thing that makes them cheap, if the provider caches it. Fireworks applies prefix-based prompt caching automatically on serverless models, discounting cached prompt tokens by around 50% by default and cutting time-to-first-token by as much as 80%. For a workload whose prompts share a large, stable prefix — the system prompt, the tool schemas, the repository context, the transcript so far — that is not a marginal optimisation. It is the difference between the agent economics working and not working.
There is an operational wrinkle here that the pricing page will not tell you, and it is the strongest practical argument in this whole comparison. A prefix cache lives on a specific node. On a shared serverless fleet your request is routed by the provider's load balancer, and if it lands somewhere that has not seen your prefix, you pay full price and full latency. Providers expose session-affinity headers to steer this, and getting them wrong is a silent, expensive regression that shows up as cost variance rather than as an error.
So the cache-hit rate for an agent is partly a property of infrastructure you do not control — until you move to a dedicated deployment, where the fleet is yours and the prefix stays put. That is a second, independent reason the ladder matters more than the entry rate, and it is discussed at length in context caching economics.
Before comparing per-token prices at all, instrument your cached-token ratio for one day of real agent traffic. A provider that is 15% more expensive per token and hits your prefix cache twice as often is the cheaper provider, and no published rate card will surface that.
The four ladders, and where each one stops
Read the four as ladders rather than as price points and they sort cleanly. Together is the only one that spans the whole range — per-token serverless, dedicated endpoints near $6.49 an hour for an H100, and raw HGX clusters from around $3.99 per GPU-hour that nobody else on this list resells. Fireworks covers the top two rungs and prices the first one under the field, at roughly $0.90 per million on 70B-class models against Together's $1.04, with similar gaps on newer open models. Baseten covers the top two as well, but its dedicated capacity is brokered across roughly twenty cloud providers rather than owned, which is a resilience and cost-arbitrage story rather than a peak-performance one. Modal has no per-token rung: you are billed for GPU time from the first request, which looks expensive at prototype scale and stops looking expensive at exactly the moment the arithmetic above kicks in.
Where a ladder stops is where a migration starts, and migrations between these platforms are not free — the model catalog, the fine-tuning artifacts, the observability integration and the rate-limit behaviour are all platform-specific. The provider whose top rung you will eventually need is worth more than a 15% discount on the rung you are standing on today. The general form of this argument is in provisioned throughput & commitments.
Cold starts are the price of the bottom rung
Once you leave always-warm serverless, scale-to-zero becomes available and cold starts become your problem. This is where the four diverge most sharply on engineering rather than on commerce. Modal built its whole product around it — a Rust-based container stack and memory snapshotting that put most launches under a second, which is the reason a per-GPU-second bill is tolerable at all. Baseten has pushed cold starts down hard with container caching, though reported figures vary widely by model size and configuration. Together and Fireworks mostly sidestep the question on their serverless tiers by keeping popular models warm for you, which is a real benefit and also the reason those tiers cost what they cost.
For agents this matters more than for batch work, because an agent's calls are bursty and interactive: a cold start does not delay a job, it stalls a trajectory that a human is watching. If you are running scale-to-zero under an agent, the cold-start number is a latency number, and it belongs in the same budget as everything else in concurrency & scaling.
Two axes that decide more than price and never appear in the comparison
Whose weights are these
If you serve stock open models, all four work and the decision is commercial. If you serve your own fine-tuned weights, the field narrows immediately: Baseten's entire proposition is putting custom and fine-tuned models into production with a real serving toolchain behind them, and Modal will run whatever container you hand it. Together and Fireworks both support fine-tuning and adapter serving, but their centre of gravity is their hosted catalog. Deciding this early is worth more than any price comparison, because it is the axis that is expensive to reverse — see fine-tuning vs RAG vs prompting for whether you should be on that path at all.
Tail latency, not median throughput
Every provider's marketing number is a median. An agent turn is a chain, and a chain multiplies tail risk: if a single call has a one-in-a-hundred chance of a very slow response, a fifteen-step trajectory contains one roughly fourteen per cent of the time. That converts a rare event into a routine one, and it means the provider comparison you should run is a P99 comparison at your own batch shape and context length, not a leaderboard read.
This is also where the dedicated rung earns its money for reasons that have nothing to do with cost: on shared serverless capacity your tail is other tenants' traffic. On a dedicated deployment your tail is your own. For teams with a latency SLO on an agent turn, that predictability is often the actual reason to move, and cost is the justification written on the ticket afterwards.
When to pick which
| Situation | Pick | Because |
|---|---|---|
| Agent workload growing fast, model choice not yet settled | Together AI | The only ladder with all three rungs, so growing out of per-token pricing is a config change rather than a migration. |
| Stock open models, per-token bill is the line item under scrutiny | Fireworks AI | Consistently under the field per token on major models, with automatic prefix caching that agents benefit from disproportionately. |
| You serve your own fine-tuned weights in production | Baseten | Built for exactly that, with dedicated capacity brokered across many clouds rather than pinned to one. |
| Custom serving stack, bursty traffic, Python-native team | Modal | Sub-second cold starts make scale-to-zero viable, and you keep full control of the engine and the container. |
If none of those describe you, the honest answer is that you are below the break-even and should stay on whichever per-token endpoint you already have, behind a gateway that keeps the switch cheap — the argument in LiteLLM vs Portkey vs Cloudflare AI Gateway vs Kong AI Gateway. Self-hosting the serving layer entirely is a further step down the same ladder, covered in vLLM vs SGLang vs TensorRT-LLM vs llama.cpp.
FAQ
At what point should an agent workload leave per-token pricing?
When sustained token throughput passes roughly 7.8 million tokens per hour, which is what one dedicated H100-hour costs at around $0.90 per million. For agent traffic that is roughly a dozen continuously running workers, far below the several hundred concurrent users a chat product would need. Measure your own tokens per hour rather than trusting the ratio.
Why do agents consume so many more tokens than chat for the same wall-clock time?
Because context is re-sent on every step. A twenty-step trajectory carrying a thirty-thousand-token working context sends that context twenty times, so prompt tokens dominate output tokens by close to two orders of magnitude. The effect grows with trajectory length, which is why long-horizon agents are the workloads that break per-token economics first.
Does prompt caching change the break-even?
Substantially, and in favour of staying serverless longer. Cached prompt tokens are typically discounted around 50%, and an agent's prefix is unusually stable, so a good hit rate can push the crossing point out considerably. The catch is that hit rate depends on request routing you do not control on a shared fleet.
Is Modal comparable to the other three at all?
Only if you are willing to own the serving stack. Together, Fireworks and Baseten hand you an endpoint for a model; Modal hands you a container runtime with very fast cold starts and expects you to bring vLLM or an equivalent. That is more work and more control, and it is the right trade when your requirements do not fit a hosted catalog.
Should I be comparing these to the frontier model APIs instead?
They answer a different question. These four host open-weight models you choose and can move between; a frontier API sells you a specific model you cannot relocate. Most production agent stacks end up using both, routed by task — the pattern described in model routing.
Further reading
On this wiki:
- Inference providers — the concept-level map of who serves models and how they bill.
- Agent cost control — where an agent's tokens actually go, step by step.
- Context caching economics — the full treatment of prefix reuse and what it is worth.
- Self-hosted inference for agents — the rung below all four of these.
- Provisioned throughput & commitments — committing to capacity without over-committing.
- vLLM vs SGLang vs TensorRT-LLM vs llama.cpp — the serving engines underneath these platforms.