Serving & access: APIs, local, gateways

E7
Concepts · The AI Model & Tooling Ecosystem

Serving & access: APIs, local inference, and gateways.

Pick an open-weight model and the posted price stops being the interesting question: as of August 2026 DeepSeek V4 Pro was listed at an identical $1.74 per million input tokens and $3.48 per million output tokens by Fireworks, Together, Baseten and Parasail. Same weights, same tariff, four vendors — so what separates them is the serving contract underneath it: the quantization you were not told about, the context ceiling and its price step, throughput and tool-call reliability under load, rate limits, and what happens to your data. Those are the variables that change your agent's behaviour quietly, which is why the thing your evaluation has to pin is the endpoint, not the model name.

Step 1Model choice and serving choice are orthogonal — and the variance has moved to the second one.

A common mistake is treating "use Llama" or "use Claude" as a single decision. You pick a model and, separately, a way to run it. For closed frontier models the second choice is nearly forced: the lab that trained it, or a cloud catalog reselling it. For open weights it is wide open, and the price convergence above is what makes it interesting — when four providers quote the same number for the same checkpoint, price has stopped being a differentiator and has become a filter you apply before looking at anything that matters.

Treat this as two decisions with two different evidence bases. The model is decided by capability on your own evaluation set, which is the argument in choosing a model. The serving path is decided by data boundary, cost at your volume, and the failure modes you can tolerate — and a change to it is a production change even when the model name in your config is untouched.

Step 2The five access options, and who each one is actually for.

First-party provider API

Call the lab that trained the model. You get the newest snapshots first, the full feature set — tool use, prompt caching, reasoning-effort controls — and their reliability engineering. Trade-off: only their models, their pricing, their roadmap. The default for closed frontier models, and the only route to features that ship before any intermediary supports them.

Cloud-hosted model catalogs

Amazon Bedrock, Google Vertex AI and Azure AI Foundry expose many models, including third-party and open-weight ones, behind an in-cloud endpoint. Value: data stays inside an existing cloud boundary, with unified billing and IAM — frequently the deciding factor under data residency and sovereignty constraints. Trade-off: model availability lags the first-party source, and you inherit the cloud's regional and quota limits.

Independent inference providers

Together, Fireworks, Baseten, Groq, Cerebras and others specialise in serving open weights, competing on latency, throughput and reliability far more than on list price. Value: the fastest route to a popular open model with no infrastructure to run. Trade-off: a third party holds your data and your uptime, the model menu is whatever they chose to host, and — per step 3 — two of them serving "the same model" may not be serving the same thing. Compared head to head in Together vs Fireworks vs Baseten vs Modal.

Local / self-hosted serving

Run open weights on hardware you control with vLLM, SGLang or TensorRT-LLM in production, or llama.cpp, Ollama and MLX on a workstation. Value: maximum control and privacy, fixed cost at high utilisation, and the only configuration where you choose the quantization. Trade-off: you own capacity planning, batching efficiency, patching and uptime — a real engineering function, costed in self-hosted inference for agents. The engine comparison is vLLM vs SGLang vs TensorRT-LLM vs Dynamo; the laptop end is Ollama vs LM Studio vs llama.cpp vs MLX.

Gateways and routers

OpenRouter, LiteLLM, Portkey and the cloud AI gateways sit in front of one or many of the above, presenting a single API and adding key management, fallback and load-balancing, spend caps, caching and unified logging — see AI gateways. Value: provider independence and operational control in one place. Trade-off: another component in the hot path to operate and secure, a little added latency, and one more party that sees every prompt.

Step 3When the price is identical, these five things are not.

This is the part that gets discovered in production. Two endpoints advertising the same checkpoint at the same price can differ on every axis that decides whether your agent works.

  • Quantization. Whether you are served the original weights or an FP8 or FP4 derivative is a serving decision, frequently undisclosed, and it moves exactly the behaviour agents depend on — long-chain arithmetic, strict schema adherence, tool-argument precision — more than it moves a benchmark average. Self-hosting is the only configuration where this is your choice; see distillation and quantization.
  • Context ceiling, and the price step inside it. Providers truncate the advertised window differently, and long context is often tiered rather than flat — OpenRouter notes MiniMax M3 prices rising above 512k tokens. Budget against the step, and remember that the usable length is shorter than the advertised one regardless of who serves it.
  • Throughput under load, not in the datasheet. OpenRouter's own June 2026 open-weight roundup warns plainly that provider quality varies for GLM 5.2 and puts cross-provider throughput at a ceiling around 78 tokens per second. For an agent, throughput is a correctness issue as much as a comfort one, because a tail-latency timeout mid-trajectory is a failed task — measuring agent latency.
  • Tool-call reliability. The same weights behind different serving stacks produce different rates of malformed tool calls, because the grammar enforcement and template handling are the host's, not the model's. OpenRouter's routing now uses tool-call telemetry to deprioritise underperforming providers, and reported GLM-5 tool-call errors falling 88% when it shipped that in March 2026 — a figure that only makes sense if the baseline variance was large. If your agent depends on tool calling or structured outputs, this is the axis to test first.
  • Rate limits and retention. Burst limits decide whether a fan-out of subagents runs or queues — rate limits and provider capacity. Retention and abuse-monitoring terms decide whether the endpoint is usable for regulated data at all, which is zero data retention rather than a pricing question.

The practical consequence is that "we run model X" is not a reproducible statement. Pin the provider, the endpoint and the quantization alongside the model version, and re-run your evaluation when a router moves you — the discipline in pinning and verification. Identical weights already produce measurably different behaviour across serving paths, which is the subject of same weights, different refusals.

Step 4The arithmetic that actually decides serverless, dedicated or self-hosted.

Per-token serving is opex that scales with traffic; dedicated capacity and self-hosting are fixed cost that only pays back at high, sustained utilisation. Work the crossover with your own numbers rather than a rule of thumb. Take an agent fleet at 2 billion input and 200 million output tokens a month, priced at the DeepSeek V4 Pro figures above:

# Serverless, per-token (Aug 2026 list: $1.74 in / $3.48 out per 1M)

2,000M input  x $1.74 / 1M  =  $3,480
  200M output x $3.48 / 1M  =  $  696
                               -------
                               $4,176 / month

# Dedicated / self-hosted: a node of 8 accelerators,
# assume $2.50 per GPU-hour on demand, run continuously

8 x $2.50 x 24 x 30         =  $14,400 / month

# Crossover is ~3.5x this traffic — and only if utilisation
# stays near continuous. Reserved rates and spot move it;
# an idle node at 3am does not get cheaper.

Three things the arithmetic hides, and all three have bitten teams that only did the arithmetic. Utilisation is the whole argument: agent traffic is bursty and a dedicated node priced at 24×7 but used during business hours costs three times what the spreadsheet said — the fixed-versus-variable trade in fixed vs variable costs, and the commitment question in provisioned throughput and commitments. Caching changes the input side by an order of magnitude on agent workloads, where the same system prompt and tool list are resent every turn, so compute it after caching, not before — prompt caching and context-caching economics. And self-hosting adds a salary line that never appears in a per-token comparison.

For most teams the right starting posture is unchanged and worth stating plainly: call the first-party API or one independent provider for whichever model passes your evaluation, behind a thin internal abstraction of your own. That seam is the cheap insurance — it lets you later add a fallback, move behind a gateway, or self-host the high-volume path without rewriting the application. Premature multi-provider routing is over-engineering; an un-abstracted hard dependency on one endpoint is under-engineering.

Decide the model on capability and your own eval. Decide the serving path on data boundary, cost at your real volume after caching, and the failure modes you can absorb. Then — the step almost everyone skips — write the provider, endpoint and quantization into the same pinned config as the model version, and put one tool-call-heavy case in your eval suite so a silent serving change shows up as a test failure rather than as a customer report. Price convergence means the cheapest provider is rarely the question; the one whose contract you can reproduce is.