Redis LangCache vs GPTCache vs Portkey vs Cloudflare AI Gateway

Every answer-reuse layer sells a hit rate, and a hit rate is a threshold setting — the number nobody publishes is the hit rate at a tolerated wrong-answer rate, which is the only form of it you can act on. Portkey documents the honest spec of this whole category by accident: its semantic cache applies only to requests under 8,191 tokens with four or fewer messages, which describes the first turn of a chat and excludes every agent loop. As of November 2026 the managed option is in public preview, the popular open-source one has no tagged release since 2024, and the edge option does exact match only.

By Agentic AI Wiki 13 min read

Portkey's documentation contains the sentence that settles this entire category, and it is not a marketing claim: its semantic cache applies only to requests under 8,191 tokens with four or fewer messages. That is a precise description of the first turn of a chat, and it excludes every agent loop you will ever run. Before you compare these four products on hit rate, notice that a hit rate is a threshold setting — the number worth having is the hit rate at a wrong-answer rate you can live with, and none of them publishes the pair.

At a glance

Four products that all answer a request without calling the model, arranged by where they sit and what they will match on. Status as of November 2026, from each project's own documentation.

ProductShapeMatchingStatus
Redis LangCache Managed service on Redis Cloud, part of Redis Iris Embedding plus a reranker, both trained for caching Public preview
GPTCache Self-hosted Python library (Zilliz) Pluggable embeddings and vector store Latest tagged release 0.1.44, Aug 2024
Portkey Feature of an LLM gateway Exact key first, then cosine similarity Shipping; semantic tier plan-gated
Cloudflare AI Gateway Feature of an edge gateway Exact match with a configurable TTL Semantic matching not shipped
Four answer-reuse layers compared A four by three matrix. Redis LangCache matches with trained embedding and reranker models, lets you set the threshold per cache, and is in public preview. GPTCache offers pluggable embeddings and in-code thresholds, but has had no tagged release since 2024. Portkey matches exact keys first then cosine similarity, documents a default threshold of 0.95, and ships as a gateway feature. Cloudflare AI Gateway offers exact-match caching only, so no threshold applies and semantic matching is not shipped. Legend: strong, partial, weak or absent. Matching, threshold control, and status What it matches on Who sets the threshold Status, Nov 2026 Redis LangCache Embedding + reranker You, per cache Public preview GPTCache Pluggable embeddings You, in code No release since 2024 Portkey Exact key, then cosine Default 0.95, eligibility caps Shipping gateway feature Cloudflare AI Gateway Exact match only Not applicable Semantic not shipped Strong Partial Weak or absent
Only two of the four will match on meaning at all, and only one of those is a product you can buy with a price on it.

The hit rate is a dial you turn, not a property you measure

Start with the number every one of these products leads with, because the number is not wrong so much as incomplete. A semantic cache embeds the incoming request, finds the nearest stored request, and serves the stored answer if the similarity clears a threshold. Move the threshold and you move the hit rate. There is no setting at which the cache is simply "accurate".

The sizes of the dial's range are visible in the products themselves, and the spread is startling for a single cosine scale. Portkey documents a default similarity threshold of 0.95. Redis's own LangCache tutorial walks through an example configuration at 0.65. Those are two vendors' defaults for the same operation, three tenths apart, which is not a disagreement about implementation — it is a disagreement about how wrong an answer is allowed to be.

Practitioner write-ups put rough numbers on the shape of the curve: around 0.97 you get high precision and a hit rate in the single digits; around 0.93 you get 20–30%; and at the looser end, several report that 3–7% of cache hits return an answer that does not actually belong to the question. Treat those figures as indicative rather than measured — they come from blogs and vendor-adjacent posts, not from an independent benchmark, and we could not trace any of them to a primary evaluation. The shape, though, is not in dispute, and a 2026 paper on the operational gap in semantic caching (arXiv:2606.19719) exists precisely because the field has been reporting single operating points instead of the trade-off across thresholds.

Which gives you the question to ask a vendor, and it is a question none of the four currently answers: what is your hit rate at a 1% false-hit rate on my traffic? Until somebody publishes that pair, a quoted "up to 90% cost reduction" is a statement about a threshold they chose, on traffic that was not yours.

Redis LangCache — the only one built as a product, and it is still in preview

LangCache is the most serious engineering effort in the group. It ships as a managed service on Redis Cloud rather than a library you operate, and it is part of the wider Redis Iris context engine, which Redis expanded on 4 November 2026 with Agent Memory at GA and Agent Mesh in beta. The piece that matters for caching is that Redis trained its own embedding and reranker models for this job rather than reusing a general-purpose embedder — a reranker in front of the similarity decision is the right architectural answer to the threshold problem, because it lets a cheap recall step be loose and a more expensive precision step be strict.

The claims are vendor claims and should be read as such: up to 90% lower API cost on the product page, with launch materials citing up to 15× faster responses on cache hits and up to 70% lower token usage, against a demo that showed 6×. The gap between "up to 15×" and "demo showed 6×" is the useful information in that sentence.

The status is the real constraint. Redis's documentation describes LangCache on Redis Cloud as a public preview, with features and behaviour subject to change. Pricing has been listed as free during the preview with general-availability pricing to be determined, on a page that is not recently dated — so you cannot currently model what this costs at steady state, which makes it difficult to put on a critical path. Unpinned vendor defaults is the relevant discipline: a preview service is one where the behaviour you measured is explicitly not promised.

GPTCache — the reference implementation, and a maintenance question

GPTCache is where most people first met this idea, and it remains the most-cited open-source semantic cache. Architecturally it is the most honest of the four: you choose the embedding model, the vector store, the similarity evaluator and the eviction policy, which means the threshold problem is visibly yours rather than hidden in a managed default.

The maintenance picture is mixed and worth stating carefully rather than dismissively. The latest tagged release on PyPI is 0.1.44, dated August 2024. The repository is not dormant — there is a "Prepare v0.2.0 release" pull request opened in September 2026, along with 2026 work on eviction policies — but we could not confirm that v0.2.0 has been tagged, and a secondary summary reports the maintainers saying the project would not add support for new APIs and models. Open issues asking for newer model support have sat unresolved since 2024.

Read it as what it is: a well-designed reference you can vendor and maintain yourself, with a two-year gap between the last release and the present. For a team that wants to own the threshold and the eviction policy, that is an acceptable trade. For a team that wants a dependency, the version history is the whole answer.

Portkey — and the spec sentence nobody meant to write

Portkey's caching is the most completely documented of the four, which is why it is the most useful. It checks an exact key first and falls back to cosine similarity, documents a default threshold of 0.95, and then states the eligibility rules: semantic caching applies only to requests under 8,191 tokens with four or fewer messages.

Those two limits are the most informative numbers in this post. 8,191 tokens is one token short of 8K, which is the context limit of a common embedding model — which reads far more like the embedder's input size leaking through the abstraction than like a product decision about who should use this. Four messages is a system prompt, a user turn, an assistant turn and one more. Together they draw an exact boundary around the workload semantic caching was built for: short, early-turn, chat-shaped requests.

There is also a documentation conflict worth knowing about before you build on it. The docs present 0.95 as a default, implying it is configurable; a Portkey-authored post about thresholds states the threshold is not user-tunable. We could not resolve which is current, and the answer determines whether you can operate this cache at all — so verify it against your own account before designing around either reading. Braintrust's write-up additionally reports the semantic tier as limited to select enterprise plans.

Cloudflare AI Gateway — exact match, and the case for honesty about it

Cloudflare's AI Gateway caches on an exact match with a configurable TTL, and most sources agree that semantic matching is roadmap rather than shipped. One third-party comparison lists it with MinHash and cosine similarity; we could not verify that against Cloudflare's own documentation, and in a category this prone to over-claiming the absence of a feature in the vendor's docs is the stronger evidence.

An exact-match cache is not a lesser version of a semantic one — it is a different product with a different risk profile and a false-hit rate of zero. For fan-out traffic where many clients send byte-identical requests, it is the correct tool and the threshold problem does not exist. For an agent loop it is almost entirely useless, because no two prompts in a trajectory are identical. The honest position is that this row of the comparison is a different column of the market.

What an agent loop does to all of this

Now put an agent in front of these four and the comparison collapses in a specific direction. An agent turn's prompt is a system prompt plus tool schemas plus a conversation plus every tool result so far. Three properties follow, and each one defeats a semantic cache for a different reason.

Three caches on one request path A left to right pipeline. The agent loop sends a turn to a gateway, which holds the exact-match and semantic caches keyed on the whole prompt. On a miss the request goes to the provider API, which holds a prefix cache keyed on the unchanged head of the prompt, and from there to the model. A changed tool result at the tail of the prompt defeats both gateway caches and leaves the provider prefix cache intact, which is why the prefix cache is the one that pays in an agent loop. Where each cache sits, and what each one is keyed on Agent loop turn n of many ask Gateway exact key match semantic match miss Provider API prefix cache on the prompt head tokens Model Appends a tool result to the tail each turn Keyed on the whole prompt, so the tail defeats it Keyed on the unchanged head, so the tail does not Pays either way for the output
The gateway caches are keyed on the whole prompt; the provider's prefix cache is keyed on its unchanged head. Only one of those survives a growing trajectory.

The prompt is too long and has too many messages, so on Portkey's documented rules it is not eligible at all, and on any embedding-based matcher it is being compressed to a single vector that is dominated by the parts that never change. The prompt is nearly unique, because the tail carries timestamps, IDs and tool output, so the near-duplicate population the cache depends on does not exist — and the closer two prompts are in embedding space, the more likely the difference between them is exactly the part that matters. And the consequence of a false hit is not a mildly wrong answer, it is a wrong action: a cached tool call executes against today's state with yesterday's arguments, which is a different severity class from a wrong chat reply.

What does pay in an agent loop is the cache one layer down. Provider prefix caching is keyed on the unchanged head of the prompt, which is precisely the part an agent repeats, and it is exact rather than approximate, so it has no false-hit rate at all. The economics are worked through in context-caching economics, and the short version is that the stable head of an agent prompt is both the largest and the most cacheable part of it. The gateway's semantic cache and the provider's prefix cache are not competing products; one of them is for chat and the other is for agents.

The exception is real but narrow: the first turn. A support agent where thousands of users open with a paraphrase of the same twenty questions has genuine near-duplicate traffic, and a semantic cache on that entry point can take a large fraction of it before any agent machinery starts. Cache the entry, not the loop.

When to pick which

SituationPickWhy
Agent trajectories, many turns None of them — use provider prefix caching Prompts are unique and long; the stable head is the cacheable part
High-volume FAQ or support entry point Redis LangCache, if you can accept a preview Reranker-backed matching is the right answer to the threshold problem
Byte-identical fan-out traffic Cloudflare AI Gateway Exact match, zero false hits, no threshold to operate
You already route through Portkey Portkey, inside the documented limits No new dependency; verify the threshold is tunable on your plan first
You need to own the threshold and the eviction policy GPTCache, vendored Every knob is yours — and so is maintaining a library last released in 2024

FAQ

Is semantic caching worth it for an agent at all?

At the entry point, often yes; inside the loop, almost never. The loop's prompts are long, nearly unique and consequential when wrong, which is the exact profile a similarity-matched cache handles worst.

What single number should I ask a vendor for?

Hit rate at a fixed false-hit rate — for example, the hit rate achievable while fewer than 1% of hits are wrong, measured on a sample of your own traffic. A hit rate quoted without its error rate is a threshold choice presented as a result.

How do I measure the false-hit rate myself?

Shadow the cache before you trust it: serve from the model, ask the cache what it would have returned, and have a judge or a human compare the two on a sample. That gives you both numbers on your own distribution, which is the only distribution that matters.

Is prompt caching the same thing as semantic caching?

No, and conflating them is the most common error here. Prompt or prefix caching is an exact match on the unchanged head of a prompt and skips recomputation inside the provider; semantic caching is an approximate match on the whole request and skips the model entirely. Different keys, different savings, different failure modes.

Does a reranker remove the threshold problem?

It improves the trade-off rather than removing it. A loose recall step followed by a strict reranker gets you more hits at the same precision, but there is still a cut-off somewhere, and it is still yours to choose and monitor.

Further reading

On this wiki:

Project sources: