Prompt caching.
The same 20,000-token system prompt gets re-read and re-billed on every single turn of a conversation — unless you cache it, in which case that span costs roughly a tenth as much and arrives faster. This entry explains the one rule that governs prompt caching (it is a prefix match), why a single stray timestamp silently destroys it, and how to tell whether yours is working.
What is actually being cached.
Before a model generates anything, it processes your entire prompt — every instruction, every retrieved document, every prior turn — into internal state. That step is real compute, and you pay for it in input tokens and in latency, on every request.
Prompt caching lets the provider keep that internal state around and reuse it when the next request starts with the same text. Nothing about the model changes; no data is "learned." It is a compute cache, and it has exactly one rule everything else follows from:
Caching is a prefix match. The cache is keyed on the exact bytes of your prompt from the beginning up to a marked point. A single byte that differs at position N invalidates the cache for everything from N onward. Content before the change is still reusable; everything after it is not.
So the design question is never "should I turn caching on." It is: is my prompt ordered so that the stable parts come first? Providers assemble the prompt in a fixed order — typically tool definitions, then the system prompt, then the conversation — and that order is the order the prefix is built in.
The economics, and when it does not pay.
Cached tokens are not free, and the asymmetry is the whole story:
- Writing to the cache costs more than a normal request — commonly around 1.25× the base input price for a short-lived entry, and roughly 2× for a longer-lived one.
- Reading from the cache costs far less — typically on the order of a tenth of base input price.
That gives a break-even: with a short time-to-live, roughly the second request already pays for the first one's write premium; with a long TTL, you need about three. Entries expire — a short TTL is measured in minutes — so the question is whether your traffic re-hits the same prefix before it lapses. A longer TTL survives quiet gaps but costs more to write, which makes it a bet on bursty traffic rather than a free upgrade.
Cases where caching is the wrong tool:
- The prompt differs from the very first token. No shared prefix, nothing to reuse — you would pay the write premium for zero reads.
- The prefix is too short. Providers set a minimum cacheable length, and it is model-specific — often somewhere between a few hundred and a few thousand tokens. Below it, nothing caches and no error is raised. This is the most common cause of "I enabled caching and nothing happened."
- Traffic is too sparse. If requests arrive further apart than the TTL, every one of them is a cache write.
The silent invalidators.
Nothing fails loudly when caching breaks — you simply keep paying full price. Each of these puts something volatile early in the prefix, and thereby invalidates everything after it:
- A timestamp or request ID in the system prompt. "Current date and time: …" at the top of a prompt makes every single request a unique prefix. If the model needs the date, put it in the latest message, not the system prompt.
- Non-deterministic serialization. Serializing a dictionary without sorting keys, or iterating an unordered set, produces different bytes for the same logical content on different runs.
- A per-user or per-session value spliced into shared instructions. Interpolating a user ID into the system prompt gives every user a private prefix and kills all cross-user sharing.
- A tool list that varies. Tool definitions render at the very front. Adding, removing, or reordering a tool mid-conversation invalidates the entire cache — sort tools deterministically and keep the set stable.
- Conditional sections.
if flag: system += …turns every combination of flags into a separate prefix, each with its own cold start. - Switching models. Caches are per-model. A "cheap model for the easy sub-task" optimization can cost more than it saves once you account for the cold cache on both sides.
The fix in every case is the same shape: stable content first, volatile content last. Freeze the system prompt, make serialization deterministic, and inject anything that changes per-request after the cached span rather than before it.
Verify it, and why agents care most.
Providers report cache activity in the response's usage figures — typically a count of tokens written to cache, a count read from cache, and a count processed at full price. That is your instrument:
- Repeated requests with an identical prefix should show a non-zero cache-read count. If it stays at zero, a silent invalidator is at work — diff the rendered prompt bytes between two consecutive requests and look for what moved.
- Do not read the full-price token count as "total prompt size." It is the uncached remainder only. An agent that ran for an hour can show a small uncached figure precisely because the rest was served from cache.
Agents are where caching stops being an optimization and becomes structural. An agent loop re-sends the entire growing transcript on every step, so a 30-step task re-processes the same instructions and tool definitions thirty times. Uncached, cost grows roughly with the square of the number of steps. This is why context engineering and cache design are the same discipline: where you put a piece of content determines not just whether the model attends to it, but what the run costs.
The takeaway: caching is not a flag you flip, it is a property of how you order your prompt. Put the frozen instructions and tool definitions first, put retrieved documents and conversation history next, put the per-request question last — then check the usage numbers to confirm the cache is actually being read. The context-caching economics deep-dive works the break-even math for real workloads, and the cost control in the loop and unit economics chapters put it alongside the other levers on cost, quality, and latency.