AI Gateways

Five jobs wearing one name.

"AI gateway", "LLM proxy" and "model router" are used interchangeably for products that do some subset of five separable jobs. Naming them separately is the whole trick, because you can buy them separately and most of the pain comes from not noticing that:

• Routing and failover. Pick a model or a provider per request, retry elsewhere when one is down or rate-limited. The mechanics — and why the escalation signal is the hard part — are in model routing.
• Key custody. One credential for your application, many provider credentials held by the gateway, so an engineer never handles an sk- key and a leaked application key can be rotated in one place.
• Caching. Exact-match or semantic reuse of responses, plus the routing needed to keep provider-side prompt caches warm.
• Policy. Rate limits per key, spend ceilings per tenant, content filters, PII redaction, and refusing calls that violate a data-residency rule.
• Metering and attribution. A normalised record of every call — model, tokens in and out, cost, latency, which user, which agent run — in one schema across providers who all report usage slightly differently.

Only the last one is genuinely hard to reproduce, and only the last one keeps getting more valuable as you grow. That is the market's read too: OpenRouter, which routes across 400-plus models from dozens of providers and charges roughly 5% on the inference spend passing through it, was reported in August 2026 to be acquired by Stripe for over $7 billion — a payments company buying the layer that counts AI usage, seven months after completing its purchase of the usage-metering vendor Metronome.

Anything in the request path is an availability dependency.

A gateway you call synchronously is not a service you use; it is a service you depend on. Three costs arrive with the hop, and only one of them is on the pricing page:

• Availability multiplies down. A 99.9% gateway in front of a 99.9% provider gives you about 99.8% — you have added a second thing that can be broken while the models are fine. That is the exact opposite of the reason most people adopt one.
• Latency adds per call, not per task. Ten to fifty milliseconds of proxy overhead disappears inside a chat response and does not disappear inside a twenty-step agent loop or a voice turn with a 300 ms budget — see cost, quality & latency.
• A percentage is a tax on steps. A 5% take rate on a chatbot is a rounding error on one call per user turn. The same rate on an agent that makes thirty model calls per task is still 5% — but 5% of a bill that is itself thirty times larger, which is why gateway spend surprises teams exactly when agent adoption works.

The part you cannot build in a week is the ledger.

Teams that self-build a proxy usually get routing and key custody working in an afternoon, then spend two quarters on the boring half. The boring half is what you are actually buying:

• Normalisation. Every provider reports usage with different field names, different cache-hit accounting, different treatment of reasoning tokens, and revises prices on their own schedule. One schema across all of them is unglamorous, endless work.
• Joining spend to outcome. A per-call cost is nearly useless; a cost joined to the agent run, the tenant and whether the task succeeded is the number that runs your business, and it is the reason cost attribution is an ops discipline rather than a dashboard.
• Enforcement, not just reporting. A ledger that can refuse the next call when a tenant is over budget is worth more than one that emails you a report — the difference between cost control and cost accounting.

Note what this implies: the ledger jobs do not require the gateway to be in your request path at all. Metering can consume the same telemetry your observability stack already collects. Splitting the two — direct provider calls for the data plane, the gateway as a control plane — gets you the ledger without the availability dependency, and is a shape too few teams consider because the products are sold as one thing.

Buy the ledger, keep the exit.

Whatever you choose, choose it in a way that survives the vendor being acquired, repriced, or degraded — all three happened somewhere in this market during 2026. Three properties are worth insisting on:

• An OpenAI-compatible surface on both sides, so the gateway is a URL and a key rather than a framework. If switching it off means rewriting call sites, it was never a proxy.
• Your own provider accounts, at least for the models you depend on. A gateway holding the only commercial relationship with your model vendor owns your capacity during a shortage, and capacity is what runs out first — see rate limits and provider capacity.
• A tested bypass. A flag that sends traffic direct, exercised on a schedule, not the first time the gateway is down.

Related: inference providers for who is on the other side of the hop, managed agent runtimes for the same buy-versus-build question one layer up, and unit economics for the number all of this feeds.