All four of these products lead with the same promise — if your provider goes down, we fail over to another one — and that promise is the weakest reason to buy any of them. Failover is not a switch that preserves behavior; it is a silent deploy of a different model, with different tool-calling semantics and different refusal behavior, executing for the first time during an incident. The durable reason to put a gateway in front of your agent is credential and budget control, and the question that actually decides which one is who gets paged when the hop itself is down.
At a glance
One endpoint in front of many providers, sold four different ways.
| Product | Who operates it | Shape | Best fit |
|---|---|---|---|
| LiteLLM | You | MIT-licensed Python proxy you deploy. | Self-hosted models beside cloud APIs, no markup. |
| Portkey | Vendor (self-host on paid tiers) | Managed gateway plus an observability product. | Governance and guardrails without running infra. |
| Cloudflare AI Gateway | Vendor | Edge proxy; a URL change, nothing to deploy. | Teams already on Cloudflare. |
| Kong AI Gateway | Your platform team | AI plugins on a Kong data plane you already run. | Enterprises with an existing Kong mesh. |
The four jobs, and which one is a trap
Strip the marketing and a gateway does four things. Three of them are unambiguous wins and are the reason to adopt one at all:
- One credential surface. Provider keys live in the gateway, not in twelve services and a notebook. Rotation becomes one operation, and revoking a team's access does not require a deploy.
- Budgets and attribution. Virtual keys per team, per customer, per environment, with hard spend caps. This is the thing most teams actually came for, and it is genuinely hard to retrofit — see cost attribution.
- Caching and a central observation point. One place where every model call is logged, priced, and optionally cached — including semantic caching, which is impractical to build per-service.
The fourth is automatic cross-provider failover, and it is oversold to the point of being misleading. It is presented as an availability feature, which invites you to reason about it as uptime arithmetic: two providers at 99.9% give you 99.9999%. That arithmetic assumes the fallback does the same job, and it does not.
The failure this produces is worse than an outage because it does not look like one. Your dashboards go green, your agents keep running, and somewhere a tool call is being emitted with arguments the other provider coerced differently — so instead of a clean error you get quiet, distributed wrongness during the exact window when nobody has spare attention. An outage is a page; a silent behavior change is a postmortem three weeks later.
None of which means don't configure failover. It means treat the fallback as a shipped configuration: run your eval suite against it on the same schedule as the primary, and route a small continuous share of live traffic through it so the path is warm and observed. That is the argument in rate limits and provider capacity, and the gateway is where you implement it.
LiteLLM — deep dive
What it is
An MIT-licensed proxy you deploy yourself, exposing an OpenAI-compatible endpoint over a hundred-plus providers, with virtual keys, per-key budgets and a dashboard. It is the default choice for teams running their own weights alongside hosted APIs, because it is the option that treats a vLLM endpoint and a frontier API as the same kind of thing.
What it buys
No markup, no third party in the data path, and no negotiation about what you are allowed to log. For anyone with data-residency constraints this is not a preference, it is the only shape that qualifies — the same reasoning as data residency and sovereignty.
What it costs
You now operate a Python service in the hot path of every model call. It needs capacity planning, upgrades, an on-call rotation, and a horizontally scaled deployment with shared state — because a per-pod rate limiter across an autoscaled fleet is not a rate limiter. Published benchmarks disagree wildly on where a single instance saturates, which is itself the finding: measure it on your own traffic shape before you believe any number, including the ones in those benchmarks.
Portkey — deep dive
What it is
A managed gateway with a governance suite around it: semantic caching, guardrails, prompt management, tracing and budgets in one place. Self-hosting exists on the paid tiers, which is the right question to ask early if you expect to need it.
What it buys
The features you would otherwise assemble from four tools, working together on day one. Semantic caching and guardrails in particular are substantial builds, and getting them as configuration rather than as roadmap is a real transfer of engineering months.
What it costs
Prompts and responses transit a third party. That is a procurement conversation, a DPA, and — for regulated workloads — sometimes a hard stop. It is also a second vendor whose incident you cannot fix, sitting in front of the first vendor whose incident you cannot fix.
Cloudflare AI Gateway — deep dive
What it is
An edge proxy in front of your existing provider calls, adding caching, rate limiting, retries, fallbacks and per-provider analytics. Adoption is a base-URL change; there is nothing to deploy and nothing to scale.
What it buys
Observability and control at close to zero adoption cost, which makes it the correct first gateway for almost anyone who wants to find out whether they need a gateway. If it turns out you needed budgets per customer and semantic caching, you have lost a week rather than a quarter.
What it costs
Its value is highest inside one ecosystem and drops outside it. It is also the option with the least control over the data path: you get the analytics the platform gives you, on the retention the platform offers.
Kong AI Gateway — deep dive
What it is
AI-specific plugins on Kong's existing API gateway — model routing, credential management, rate limiting, prompt policy — applied by the same data plane and the same control surface as your REST traffic.
What it buys
One operational story. The platform team already runs Kong, already has a policy engine, already has audit and RBAC wired up; LLM traffic stops being a special case that needs its own runbook and its own reviewer. In an enterprise, that is frequently worth more than any individual feature on the comparison grid.
What it costs
LLM-specific capability trails the dedicated products, and it will keep trailing — Kong's centre of gravity is API management. If you want semantic caching, prompt versioning and eval integration as first-class features, you are asking a general-purpose gateway to be a specialist one.
Cross-cutting comparison
Who operates the hop is the irreversible decision
Feature parity in this category arrives fast — semantic caching and guardrails were differentiators a year ago and are table stakes now. What does not converge is the operational and legal shape: whether a third party sees your prompts, whether your team carries the pager, whether the thing is governed by the platform org or by the product team. Choose on that axis and the feature comparison mostly resolves itself.
You are adding a single point of failure in front of a single point of failure
This is the objection nobody puts on their landing page. Every model call now depends on both the provider and the gateway, so the gateway must be at least as available as what it protects — otherwise the multi-provider resilience you bought is strictly negative. Concretely: the client SDK needs a direct-to-provider bypass path, tested, so an operator can route around the gateway without a code change. If the gateway is the only way to reach a model, you have centralised your risk and called it redundancy.
Latency overhead: measure it, do not read it
Published figures for these same products disagree by an order of magnitude — the same gateway is described as sub-millisecond in one comparison and tens of milliseconds in another. Both can be true, because overhead depends on colocation, connection reuse, whether responses stream, and whether caching or guardrail checks are enabled. Against a model call measured in seconds, single-digit milliseconds are irrelevant; a synchronous guardrail pass is not. Benchmark with your own payload shapes and your own feature set enabled, and treat every published number — including the ranges in that sentence — as a hypothesis.
When to pick which
| Situation | Pick | Why | Watch out for |
|---|---|---|---|
| Not sure you need a gateway yet | Cloudflare | A base-URL change buys visibility. | Value concentrated in one ecosystem. |
| Self-hosted models beside hosted APIs | LiteLLM | Treats both as the same endpoint; no markup. | You operate it, scale it, and get paged. |
| Data cannot transit a third party | LiteLLM | The only fully self-owned option by default. | Shared state for limits across the fleet. |
| Guardrails and governance, small platform team | Portkey | Buys engineering months as configuration. | Procurement, DPA, second vendor in the path. |
| Enterprise already running Kong | Kong | One policy engine, one runbook, one auditor. | LLM features trail the specialists. |
| Chasing five-nines with two providers | Reconsider | The fallback changes behavior, not just uptime. | Silent wrongness beats a clean outage to nobody. |
FAQ
Do I need an AI gateway at all?
Not until provider keys are spread across several services, or you cannot answer "what did team X spend last month". Those two problems are what a gateway genuinely solves. One application calling one provider does not need a hop in front of it.
Isn't automatic failover to a second provider strictly better than nothing?
Not strictly. Failover routes traffic onto a model with different tool-calling semantics, different refusal behavior and a cold prompt cache, during an incident, without an eval having covered that path. Configure it, but evaluate the fallback on the same schedule as the primary and send it a trickle of live traffic so it is warm.
Does the gateway add meaningful latency?
Proxy overhead is usually negligible against a model call measured in seconds. Synchronous features — guardrail checks, semantic-cache lookups — are not negligible. Benchmark with the features you will actually enable rather than trusting a vendor's bare-proxy number.
Can I self-host all of these?
LiteLLM is MIT-licensed and fully self-hostable. Kong runs on your own data plane, with some AI capability tiered. Portkey offers self-hosting on paid tiers. Cloudflare AI Gateway is a managed service by nature. If self-hosting is a hard requirement, that constraint alone narrows the field to two.
What about OpenRouter?
OpenRouter is a hosted routing marketplace rather than a gateway you place in your own architecture — it resells access across models on its own terms. Useful for breadth of model access and for experimentation; a different answer to the credential, budget and policy questions this post is about.
What happens when the gateway itself goes down?
Every model call fails, which is why a tested direct-to-provider bypass in the client is not optional. Ask any vendor for their gateway's own availability history, and make sure your architecture can route around it without a deploy.
Further reading
On this wiki:
- Rate limits and provider capacity — the admission control this gateway should be enforcing.
- Rollout, versioning and pinning — why the fallback needs a dated snapshot too.
- Model routing — choosing per request, not per outage.
- Semantic caching — the feature most worth buying rather than building.
- Cost attribution — the problem virtual keys exist to solve.