Semantic caching.
Every other cache in your stack can only be slow or absent; a semantic cache can be wrong — it decides a hit by similarity score rather than by equality, so a near-miss returns a confident, fluent answer to a question nobody asked. That single property should reorder your priorities: it is not a caching feature, it is a retrieval system with a false-positive budget, and it needs the same evaluation, versioning and scoping you would give a search index.
Four things are called "caching" and only one of them can lie.
The word collapses four mechanisms that fail in completely different ways. Sort them before you argue about which to turn on.
- Prefix / prompt caching — provider-side reuse of the processed prompt prefix. The key is an exact token-prefix match, so a miss costs you nothing but the normal price. It cannot serve a wrong answer. This is prompt caching, and it is the one to turn on first.
- Exact-match response caching — you hash the fully-resolved request (prompt, model, parameters) and store the response. Also incapable of lying; also modest, because natural-language requests rarely repeat byte-for-byte. Normalising whitespace and casing before hashing usually doubles the hit rate for free.
- Tool-result caching — memoising the expensive tool behind a deterministic key: an embedding for a document that has not changed, a geocode lookup, a schema fetch. Frequently the biggest win in an agent and almost always the least discussed one, because the cost being avoided is not a model call at all.
- Semantic caching — you embed the incoming request, search a vector store of previous requests, and if the nearest neighbour is above a similarity threshold you return its answer. Skips the model call entirely, which is why it is attractive, and it is the only one of the four that can be confidently incorrect.
Prefix caching and semantic caching are not competitors and not substitutes. The first reduces what you pay per call; the second removes the call. Turn on the first everywhere, and treat the second as a feature you ship deliberately to a specific slice of traffic.
The threshold is where the whole design lives.
A semantic cache is nearest-neighbour search with the answer stapled on. Everything you know about embeddings applies, including the parts that hurt: cosine similarity measures topical resemblance, not equivalence of meaning, and the distinctions that most often flip an answer are exactly the ones embeddings compress away.
- Negation. "Which of these plans includes overage billing?" and "which of these plans does not include overage billing?" sit close together in embedding space and have opposite answers.
- Entities and numbers. Swapping one account ID, product name, quarter or version number barely moves the vector, and completely changes the correct response.
- Implicit context. Two identical follow-up questions ("and the second one?") mean different things in different conversations. The embedded text does not carry the conversation that disambiguates it.
Practitioners typically land on cosine thresholds somewhere in the 0.85–0.95 band, and reported production hit rates on repetitive traffic like FAQ and support workloads commonly fall between 30% and 70%. Treat both as starting coordinates, not settings: the right threshold is a function of your embedding model and your traffic, and it moves the moment you change either. Too permissive and you serve the wrong answer to the edge cases; too strict and the hit rate collapses back toward what exact matching already gave you.
The cheap structural fix is to stop trusting the threshold alone. Restrict the cache to a domain where questions genuinely repeat, and use the cached entry as a candidate — a small model can verify "does this stored answer actually answer this question?" for a fraction of the price of the full call. That turns an unbounded similarity gamble into a bounded one.
The cache key is bigger than the question.
Most semantic-cache incidents are not threshold-tuning failures. They are scoping failures: two requests that looked alike were never allowed to share an answer in the first place. Everything that could change the correct response has to be part of the key, not part of the hope.
- Tenant, user and permissions. If retrieval is permission-filtered, so is the answer. A cache shared across users is a mechanism for showing one customer another customer's data with no bug in your authorization layer at all — see identity and permissions.
- Model, version and parameters. An entry produced by a model you have since replaced is stale in a way no time-to-live catches.
- Prompt version and tool set. You changed the system prompt to fix a behaviour; the cache keeps serving the old behaviour to the requests that need the fix most.
- Corpus version. For anything retrieval-augmented, the answer is a function of documents that change. Either key on a corpus revision or accept that your cache is a mirror of last week's knowledge base.
- Locale. Multilingual embedding models place a question and its translation close together — which is convenient for search and wrong for caching if the answer must be in the asker's language.
Two more rules follow from the same logic. Never cache anything with a personalised or time-dependent answer ("what's my balance", "what happened yesterday") no matter how repetitive the phrasing. And give every entry a time-to-live short enough that a bad one ages out before someone has to page you about it.
Ship it in shadow mode, and measure precision — not hit rate.
Hit rate is the number that gets reported and the number that misleads. It is trivially maximised by lowering the threshold, and every point you buy that way is bought with wrong answers. The metric that matters is hit precision: of the requests you served from cache, what fraction would have received an equivalent answer from the model?
- Run it in shadow first. Log what the cache would have returned while still calling the model, then compare the two. A day of that gives you a real precision curve against threshold, and it costs nothing but log volume.
- Score the disagreements, don't eyeball them. A judge model comparing cached answer to live answer over a few hundred shadow hits is a normal eval, and it is how you pick the threshold defensibly.
- Put cache status in the trace. Hit or miss, similarity score, and the key of the entry served. A wrong answer that came from cache and a wrong answer the model generated need completely different fixes, and without this field you cannot tell them apart — one more thing observability pays for.
- Watch precision over time, not once. Traffic drifts, the corpus changes, and a threshold that was right at launch quietly stops being right. Re-run the shadow comparison on a schedule.
Order of operations, and it rarely varies: turn on prefix caching, add exact-match caching on a normalised key, then memoise the expensive deterministic tools. Those three are free of correctness risk and in most systems they capture the majority of the available savings. Only then consider a semantic cache, only on a narrow slice of genuinely repetitive traffic, keyed by tenant and corpus version, and only after a shadow run has shown you its precision at the threshold you plan to use. If you cannot measure that precision, you are not caching — you are sampling a distribution of answers and hoping.
Related: agent cost control for where caching sits among the cost levers, small & local models for the cheap verifier this page keeps asking for, and cost control at the loop level for the operational view.