Portkey's documentation contains the sentence that settles this entire category, and it is not a marketing claim: its semantic cache applies only to requests under 8,191 tokens with four or fewer messages. That is a precise description of the first turn of a chat, and it excludes every agent loop you will ever run. Before you compare these four products on hit rate, notice that a hit rate is a threshold setting — the number worth having is the hit rate at a wrong-answer rate you can live with, and none of them publishes the pair.
At a glance
Four products that all answer a request without calling the model, arranged by where they sit and what they will match on. Status as of November 2026, from each project's own documentation.
| Product | Shape | Matching | Status |
|---|---|---|---|
| Redis LangCache | Managed service on Redis Cloud, part of Redis Iris | Embedding plus a reranker, both trained for caching | Public preview |
| GPTCache | Self-hosted Python library (Zilliz) | Pluggable embeddings and vector store | Latest tagged release 0.1.44, Aug 2024 |
| Portkey | Feature of an LLM gateway | Exact key first, then cosine similarity | Shipping; semantic tier plan-gated |
| Cloudflare AI Gateway | Feature of an edge gateway | Exact match with a configurable TTL | Semantic matching not shipped |
The hit rate is a dial you turn, not a property you measure
Start with the number every one of these products leads with, because the number is not wrong so much as incomplete. A semantic cache embeds the incoming request, finds the nearest stored request, and serves the stored answer if the similarity clears a threshold. Move the threshold and you move the hit rate. There is no setting at which the cache is simply "accurate".
The sizes of the dial's range are visible in the products themselves, and the spread is startling for a single cosine scale. Portkey documents a default similarity threshold of 0.95. Redis's own LangCache tutorial walks through an example configuration at 0.65. Those are two vendors' defaults for the same operation, three tenths apart, which is not a disagreement about implementation — it is a disagreement about how wrong an answer is allowed to be.
Practitioner write-ups put rough numbers on the shape of the curve: around 0.97 you get high precision and a hit rate in the single digits; around 0.93 you get 20–30%; and at the looser end, several report that 3–7% of cache hits return an answer that does not actually belong to the question. Treat those figures as indicative rather than measured — they come from blogs and vendor-adjacent posts, not from an independent benchmark, and we could not trace any of them to a primary evaluation. The shape, though, is not in dispute, and a 2026 paper on the operational gap in semantic caching (arXiv:2606.19719) exists precisely because the field has been reporting single operating points instead of the trade-off across thresholds.
Which gives you the question to ask a vendor, and it is a question none of the four currently answers: what is your hit rate at a 1% false-hit rate on my traffic? Until somebody publishes that pair, a quoted "up to 90% cost reduction" is a statement about a threshold they chose, on traffic that was not yours.
Redis LangCache — the only one built as a product, and it is still in preview
LangCache is the most serious engineering effort in the group. It ships as a managed service on Redis Cloud rather than a library you operate, and it is part of the wider Redis Iris context engine, which Redis expanded on 4 November 2026 with Agent Memory at GA and Agent Mesh in beta. The piece that matters for caching is that Redis trained its own embedding and reranker models for this job rather than reusing a general-purpose embedder — a reranker in front of the similarity decision is the right architectural answer to the threshold problem, because it lets a cheap recall step be loose and a more expensive precision step be strict.
The claims are vendor claims and should be read as such: up to 90% lower API cost on the product page, with launch materials citing up to 15× faster responses on cache hits and up to 70% lower token usage, against a demo that showed 6×. The gap between "up to 15×" and "demo showed 6×" is the useful information in that sentence.
The status is the real constraint. Redis's documentation describes LangCache on Redis Cloud as a public preview, with features and behaviour subject to change. Pricing has been listed as free during the preview with general-availability pricing to be determined, on a page that is not recently dated — so you cannot currently model what this costs at steady state, which makes it difficult to put on a critical path. Unpinned vendor defaults is the relevant discipline: a preview service is one where the behaviour you measured is explicitly not promised.
GPTCache — the reference implementation, and a maintenance question
GPTCache is where most people first met this idea, and it remains the most-cited open-source semantic cache. Architecturally it is the most honest of the four: you choose the embedding model, the vector store, the similarity evaluator and the eviction policy, which means the threshold problem is visibly yours rather than hidden in a managed default.
The maintenance picture is mixed and worth stating carefully rather than dismissively. The latest tagged release on PyPI is 0.1.44, dated August 2024. The repository is not dormant — there is a "Prepare v0.2.0 release" pull request opened in September 2026, along with 2026 work on eviction policies — but we could not confirm that v0.2.0 has been tagged, and a secondary summary reports the maintainers saying the project would not add support for new APIs and models. Open issues asking for newer model support have sat unresolved since 2024.
Read it as what it is: a well-designed reference you can vendor and maintain yourself, with a two-year gap between the last release and the present. For a team that wants to own the threshold and the eviction policy, that is an acceptable trade. For a team that wants a dependency, the version history is the whole answer.
Portkey — and the spec sentence nobody meant to write
Portkey's caching is the most completely documented of the four, which is why it is the most useful. It checks an exact key first and falls back to cosine similarity, documents a default threshold of 0.95, and then states the eligibility rules: semantic caching applies only to requests under 8,191 tokens with four or fewer messages.
Those two limits are the most informative numbers in this post. 8,191 tokens is one token short of 8K, which is the context limit of a common embedding model — which reads far more like the embedder's input size leaking through the abstraction than like a product decision about who should use this. Four messages is a system prompt, a user turn, an assistant turn and one more. Together they draw an exact boundary around the workload semantic caching was built for: short, early-turn, chat-shaped requests.
There is also a documentation conflict worth knowing about before you build on it. The docs present 0.95 as a default, implying it is configurable; a Portkey-authored post about thresholds states the threshold is not user-tunable. We could not resolve which is current, and the answer determines whether you can operate this cache at all — so verify it against your own account before designing around either reading. Braintrust's write-up additionally reports the semantic tier as limited to select enterprise plans.
Cloudflare AI Gateway — exact match, and the case for honesty about it
Cloudflare's AI Gateway caches on an exact match with a configurable TTL, and most sources agree that semantic matching is roadmap rather than shipped. One third-party comparison lists it with MinHash and cosine similarity; we could not verify that against Cloudflare's own documentation, and in a category this prone to over-claiming the absence of a feature in the vendor's docs is the stronger evidence.
An exact-match cache is not a lesser version of a semantic one — it is a different product with a different risk profile and a false-hit rate of zero. For fan-out traffic where many clients send byte-identical requests, it is the correct tool and the threshold problem does not exist. For an agent loop it is almost entirely useless, because no two prompts in a trajectory are identical. The honest position is that this row of the comparison is a different column of the market.
What an agent loop does to all of this
Now put an agent in front of these four and the comparison collapses in a specific direction. An agent turn's prompt is a system prompt plus tool schemas plus a conversation plus every tool result so far. Three properties follow, and each one defeats a semantic cache for a different reason.
The prompt is too long and has too many messages, so on Portkey's documented rules it is not eligible at all, and on any embedding-based matcher it is being compressed to a single vector that is dominated by the parts that never change. The prompt is nearly unique, because the tail carries timestamps, IDs and tool output, so the near-duplicate population the cache depends on does not exist — and the closer two prompts are in embedding space, the more likely the difference between them is exactly the part that matters. And the consequence of a false hit is not a mildly wrong answer, it is a wrong action: a cached tool call executes against today's state with yesterday's arguments, which is a different severity class from a wrong chat reply.
What does pay in an agent loop is the cache one layer down. Provider prefix caching is keyed on the unchanged head of the prompt, which is precisely the part an agent repeats, and it is exact rather than approximate, so it has no false-hit rate at all. The economics are worked through in context-caching economics, and the short version is that the stable head of an agent prompt is both the largest and the most cacheable part of it. The gateway's semantic cache and the provider's prefix cache are not competing products; one of them is for chat and the other is for agents.
The exception is real but narrow: the first turn. A support agent where thousands of users open with a paraphrase of the same twenty questions has genuine near-duplicate traffic, and a semantic cache on that entry point can take a large fraction of it before any agent machinery starts. Cache the entry, not the loop.
When to pick which
| Situation | Pick | Why |
|---|---|---|
| Agent trajectories, many turns | None of them — use provider prefix caching | Prompts are unique and long; the stable head is the cacheable part |
| High-volume FAQ or support entry point | Redis LangCache, if you can accept a preview | Reranker-backed matching is the right answer to the threshold problem |
| Byte-identical fan-out traffic | Cloudflare AI Gateway | Exact match, zero false hits, no threshold to operate |
| You already route through Portkey | Portkey, inside the documented limits | No new dependency; verify the threshold is tunable on your plan first |
| You need to own the threshold and the eviction policy | GPTCache, vendored | Every knob is yours — and so is maintaining a library last released in 2024 |
FAQ
Is semantic caching worth it for an agent at all?
At the entry point, often yes; inside the loop, almost never. The loop's prompts are long, nearly unique and consequential when wrong, which is the exact profile a similarity-matched cache handles worst.
What single number should I ask a vendor for?
Hit rate at a fixed false-hit rate — for example, the hit rate achievable while fewer than 1% of hits are wrong, measured on a sample of your own traffic. A hit rate quoted without its error rate is a threshold choice presented as a result.
How do I measure the false-hit rate myself?
Shadow the cache before you trust it: serve from the model, ask the cache what it would have returned, and have a judge or a human compare the two on a sample. That gives you both numbers on your own distribution, which is the only distribution that matters.
Is prompt caching the same thing as semantic caching?
No, and conflating them is the most common error here. Prompt or prefix caching is an exact match on the unchanged head of a prompt and skips recomputation inside the provider; semantic caching is an approximate match on the whole request and skips the model entirely. Different keys, different savings, different failure modes.
Does a reranker remove the threshold problem?
It improves the trade-off rather than removing it. A loose recall step followed by a strict reranker gets you more hits at the same precision, but there is still a cut-off somewhere, and it is still yours to choose and monitor.
Further reading
On this wiki:
- Semantic caching — the concept, and what it can and cannot key on.
- Prompt caching — the other cache, and the one that helps an agent.
- Context-caching economics — the arithmetic of what a cache discount is actually worth.
- Fewer tokens versus cheaper tokens — where caching sits among the other cost levers.
- AI gateways — what else the gateway layer is doing while it caches.
- Cost control at the loop level — the ceilings that matter more than any cache.