LiteLLM vs Portkey vs Helicone vs OpenRouter: in the path, or beside it

At a glance

Four products that occupy genuinely different positions, described the way their own documentation describes them.

OpenRouter's fee structure deserves one line of clarification because it is routinely misdescribed: it does not mark up provider token rates. What it charges is a fee on money entering the system — 5.5% on card top-ups, 5% on crypto, with a floor of $0.80 per transaction — plus a 5% fee on bring-your-own-key traffic above a monthly free allowance that recent write-ups put at either the first million BYOK requests or roughly $25,000 of list-price inference, rising substantially on enterprise plans. Check the current terms before you model it; the structure is stable, the thresholds move.

Question one: in the path, or beside it

Helicone's own docs put the two integrations side by side — proxy versus async — and state the trade honestly: async keeps the logging off the critical path, so an outage or a network problem at the vendor cannot affect your application, but async "cannot offer the same suite of tools as the proxy", because the proxy is what sits on the edge and gatekeeps the request. Caching, rate limiting, API-key management, threat detection and moderation all live on the proxy side of that line.

That framing generalises to all four products and it is worth stating as a rule: anything that changes a request requires being in the request path, and everything in the request path is a dependency you have to survive. There is no configuration that gets you both. Teams discover this the hard way when they buy a gateway for its observability, deploy it as a proxy because that is the quick-start path, and thereby put a startup's uptime in front of every model call they make.

Which side you want depends on what you are actually buying:

• Buying visibility? Go off-path. Traces, cost attribution and latency histograms all work fine from an async logger, and you keep a failure domain you already understand. See tracing & observability.
• Buying control? You are in the path, so budget for it: health checks, a documented bypass, and a tested answer to "the gateway is returning 503, what happens to running agents". Graceful degradation is the page to read before, not after.
• Buying both? Run the proxy, but make the bypass a first-class code path that you exercise on a schedule rather than a comment in the runbook.

Question two: who holds the provider credential

The credential looks like a configuration detail and behaves like an architectural commitment. Whoever holds it owns the provider relationship: the rate limits, the quota increases, the abuse escalation, the enterprise agreement, and the ability to call the provider directly when the layer in between stops working.

Compare the three positions on the day something breaks. If you self-host LiteLLM or Portkey and your own gateway falls over, you point your SDK at the provider and lose your routing policy — an incident, not an outage. If you use a managed control plane with BYOK, the same is true with one more phone call. If you are on OpenRouter's credit model and OpenRouter is unavailable, your fallback is a provider account you do not have, with no quota history and no negotiated limits; opening one takes minutes for the signup and days for the rate limits to matter.

That is not an argument against OpenRouter, which is genuinely the fastest way to reach 200-plus models behind one key and the right answer for a large class of work. It is an argument for pricing the dependency honestly, and for keeping a dormant direct account on your two most important models if the agent is doing anything a business depends on.

The credential also decides where your prompts go. A self-hosted proxy keeps them inside your perimeter, which is the whole reason regulated teams reach for LiteLLM; a managed gateway sees them by construction. That is a data residency question, and it does not have a technical workaround — only a deployment choice.

The agent multiplier

Everything above applies to a chat application. Agents change the magnitudes, and in three places they change them enough to flip the decision.

Latency compounds, and the tail is what you feel

A chat turn is one model call, so an extra 30ms of proxy hop is invisible next to a two-second generation. A twenty-five step agent task pays it twenty-five times, and — more importantly — takes the gateway's p99 twenty-five times. If any single step failing kills the task, a 1% per-call failure rate gives you roughly a 22% task failure rate. Median latency is the number gateways advertise; tail behaviour under your own concurrency is the number that decides whether your agent finishes, and only a load test tells you it.

A percentage is a per-step tax

Fees that read as small on a per-request basis are levied per model call, not per user action. The relevant unit for an agent is cost per successfully completed task, and a task is dozens of calls. This is the same shape as the argument we made when Stripe bought the metering layer rather than the routing layer: whoever sits in the request path of an agent is charging per step of a loop whose step count you do not fully control.

Provider prompt caching is the one to guard

This is the trap that costs real money and generates no alert. Agents re-send a growing transcript on every step, so provider-side prompt caching is often the single largest lever on the bill — and it depends on an exactly stable prefix and correct cache-control handling. A gateway that normalises headers, reorders system content, injects its own preamble, or load-balances the same conversation across providers can silently reduce your cache hit rate. Nothing errors. The bill goes up by a multiple, attributed to "more usage".

Test it directly before you commit: run the same twenty-step task through the gateway and around it, and compare the cached-token counts the provider reports. If the gateway costs you cache hits, no amount of gateway-level caching makes that back.

When to pick which

The combination worth naming explicitly, because it is what a lot of mature agent teams end up running: a self-hosted proxy for the production path, an async logger for the analytics that must not add a dependency, and a managed marketplace key kept for evaluation work where reaching a new model in five minutes is worth more than the fee.

Further reading

On this wiki:

• AI gateways — what the layer is and what it is not.
• Model routing — the policy the gateway executes.
• Prompt caching — why an agent's stable prefix is the biggest cost lever you own.
• Graceful degradation & fallback — the plan for the day the path is broken.
• Cost attribution & budgets — the reporting the gateway position makes possible.

Project sources:

• Helicone — Proxy vs Async integration.
• LiteLLM and Helicone repositories.
• Portkey — gateway and control-plane documentation.
• OpenRouter docs — current fee schedule and BYOK thresholds.