--- title: Prompt caching cheatsheet: building cost-effective AI agents date: 2026-08-17 tags: AI Toolsets canonical_url: https://coolhandlabs.com/updates/llm-prompt-caching-what-it-is-and-key-terms-explained.md index_url: https://coolhandlabs.com/updates.md feed_url: https://coolhandlabs.com/updates.rss --- ![Prompt caching cheatsheet: building cost-effective AI agents](https://storage.googleapis.com/coolhand-public/og-images/llm-prompt-caching-what-it-is-and-key-terms-explained.png) # Prompt caching cheatsheet: building cost-effective AI agents *Cache write, cache read, refresh, breakpoint, TTL, storage fee. Six words that mean different things at three providers — and one of them charges you while nothing is happening.* --- Prompt caching is the closest thing to free money in the inference stack, and it is described almost identically by every provider that sells it: send the same context twice, pay less the second time. That description is true. It is also thin enough that teams routinely turn caching on, watch their bill go *up*, and can't say why. The reason is vocabulary. Every provider uses the same six words for six meaningfully different mechanisms. This piece is the glossary — what each term means, where the providers diverge, and which ones have a price attached. It's the foundation for the four pieces that follow: [the pricing rules](/updates/llm-prompt-caching-pricing-guide-and-cost-breakdown), and then a deep dive each on [Anthropic](/updates/llm-prompt-caching-anthropic-how-to-and-pro-tips), [OpenAI](/updates/llm-prompt-caching-openai-how-to-and-pro-tips), and [Gemini](/updates/llm-prompt-caching-gemini-how-to-and-pro-tips). ## What caching actually does > **In one line:** the model stores the computed state of a prompt prefix so it doesn't have to recompute it next time. That "recompute" step isn't free, and it isn't small. Input tokens are billed on every single request, and for most production AI workloads — agents especially — the prompt dwarfs the response: system instructions, tool definitions, and accumulated conversation or tool-call history can run into the tens of thousands of tokens, resent in full on every turn. An agent looping through a multi-step task might resend the same 50,000-token prefix a dozen times in a single run. Without caching, you pay full input price for that prefix every single time. With it, everything after the first request drops to roughly a tenth of the cost. At agent scale, that's rarely a rounding error — it's often the single largest lever available for cutting inference spend. When you send a request, the model processes your entire prompt from the first token. Caching lets the provider keep the intermediate state — the key-value tensors — for some leading portion of that prompt, so a later request that starts with the identical bytes can resume from that point instead of starting over. ## What can be cached? Two key things to remember Two consequences fall straight out of that, and both matter more than most of the guidance you'll read. **Caching only ever discounts input.** No provider's caching touches output token cost. If your workload is output-heavy — long generations from short prompts — caching cannot help you much no matter how well you implement it. Any savings projection that applies the cache discount to your whole bill is wrong by whatever fraction of your spend is output. Anthropic's docs are explicit that [prompt caching has no effect on output token generation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching); OpenAI says the same, adding that [the model still computes a fresh response](https://developers.openai.com/api/docs/guides/prompt-caching) from the cached prefix, so identical requests still aren't guaranteed identical outputs. **It's a prefix cache, not a substring cache.** The match runs from the very start of the prompt forward. Change one token near the beginning and everything after it is a miss, regardless of how much of the rest is identical. This is why "put static content first, variable content last" is the one piece of advice every provider gives. That second point is why your **system prompt** is usually the single best thing to cache. It's typically the largest block of genuinely static content in the request — instructions, persona, output format, tool definitions — and it's identical across every call, which is exactly what a prefix cache rewards. Anthropic builds its cache ordering around this directly: prefixes are assembled as `tools`, then `system`, then `messages`, and its own guidance is to ["cache stable, reusable content like system instructions, background information, large contexts, or frequent tool definitions"](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) — in practice, that means the `cache_control` breakpoint usually lands on the last block of the system prompt. OpenAI's advice points the same direction: ["place static content like instructions and examples at the beginning of your prompt, and put variable content, such as user-specific information, at the end"](https://developers.openai.com/api/docs/guides/prompt-caching), with system instructions treated as exactly the kind of repetitive, cacheable content the default breakpoint is built to capture. > **The practical upshot:** if you only cache one thing, cache your system prompt. It's the part that's stable in an agent loop, a chatbot, and a batch pipeline alike — and it's usually where the most tokens are sitting doing nothing but repeating themselves. ## When caching can save you big money Caching helps most exactly where large, mostly-static context gets resent over and over. A few patterns worth watching for: - **Agent and orchestrator loops.** Any system that makes multiple LLM calls per task — planning steps, tool calls, sub-agents — resends its system prompt and tool definitions on every step. Cache that prefix once and every subsequent call in the loop reads it instead of paying full price. - **Long-running chat conversations.** Every turn in a chat resends the entire history. Without caching, a 50-turn conversation reprocesses the earlier 49 turns 49 times over; with it, only the newest turn is billed at full rate. - **High-volume batch and extraction jobs.** Thousands of documents run through the same instruction block and schema is close to a caching best case — one write, thousands of near-free reads. - **RAG and shared-context systems.** Multiple users or requests querying the same retrieved documents, knowledge base, or system instructions can share a single cache entry, which is where the savings compound fastest. The mechanics — and which of these patterns work best on which provider — vary enough to warrant their own treatment: see [pricing 101](/updates/llm-prompt-caching-pricing-guide-and-cost-breakdown) for the cross-provider cost model, and the deep dives on [Anthropic](/updates/llm-prompt-caching-anthropic-how-to-and-pro-tips), [OpenAI](/updates/llm-prompt-caching-openai-how-to-and-pro-tips), and [Gemini](/updates/llm-prompt-caching-gemini-how-to-and-pro-tips) for provider-specific strategy. ## The billing vocabulary > **Cost shape:** three of these six terms have a price attached. Two of those are per-request. One is per-hour. **Cache write.** Creating a new cache entry. Anthropic and current OpenAI models both charge a premium for this — you pay *more* than uncached input for the privilege of storing it. Google does not charge a write premium; you're [billed at the standard input rate](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/context-cache/context-cache-overview) for the tokens used to create the cache. **Cache read** (also *cache hit*). Reusing an existing entry. This is where the savings live, and all three providers converge on a 90% discount off their base input rate. **Refresh.** A read that also resets the entry's expiry clock. Free at Anthropic and OpenAI, and the single most important mechanic in the whole subject: a cache under steady traffic never expires and never needs rewriting. Google's explicit caches don't refresh — they expire on a wall clock regardless of use. **Breakpoint.** The marker saying "cache everything up to here." The prefix ends at the breakpoint; anything after it is uncached and free to change. Breakpoints themselves cost nothing — [Anthropic is explicit](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) that adding more doesn't increase your bill, and [so is OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching). You're charged for tokens written, not markers placed. **TTL** (time to live). How long the entry survives. The word is doing very different work at each provider — enough that it gets its own section below. **Storage fee.** A per-token, per-hour rent charged for holding a cache resource, whether or not anything reads it. Only Google charges this, and only on explicit caching. It is the one line item in this entire subject that accrues while your application is idle. ## Implicit, explicit, automatic: three words, six meanings This is the vocabulary trap that causes the most confusion, because the same adjective points at different things depending on whose docs you're in. At **Google**, *implicit* means the service opportunistically matches prefixes with no code from you and no guarantee of a hit — it's [on by default for Gemini 2.5 and newer](https://ai.google.dev/gemini-api/docs/caching) and carries no storage fee. *Explicit* means you create a named cache resource with a TTL, reference it by ID, and pay rent. Google's own framing is that [explicit caching is the one that guarantees savings](https://ai.google.dev/gemini-api/docs/generate-content/caching) — implicit is best-effort. At **OpenAI**, *implicit* means the service places a breakpoint on the latest user or tool message for you. *Explicit* means you place breakpoints yourself with `prompt_cache_breakpoint`. Both write real cache entries and both are billable on current models; the difference is who chooses the boundary, not whether you get a guarantee. At **Anthropic**, the parallel term is *automatic caching* — a single top-level `cache_control` field that puts the breakpoint on the last cacheable block and advances it as a conversation grows. The alternative is *explicit cache breakpoints* placed on individual blocks. So "explicit caching" at Google is a managed resource with rent. At OpenAI and Anthropic it's a marker in your request. ## When the clock starts > **The trap:** "5-minute TTL" does not mean you have five minutes of thinking time between turns. **Anthropic** measures the lifetime [from the start of the request that writes or reads the entry](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) — not from the end of the response. Generation time is inside your window. Their own worked example: a response that takes four minutes to stream leaves roughly one minute before the follow-up must begin. Reads refresh the entry, so the constraint is really *start-to-start* gap between consecutive requests, not the gap between one response finishing and the next beginning. Instrument the wrong one and your cache-hit model will be wrong. **OpenAI** on GPT-5.6 and later runs a 30-minute lifetime that [begins when the prefix is written and refreshes whenever it's reused](https://developers.openai.com/api/docs/guides/prompt-caching). On earlier models it's an idle timer instead — roughly 5 to 10 minutes of inactivity under the in-memory policy, with a hard ceiling. **Google's** explicit caches are wall-clock resources. You set a TTL at creation (it [defaults to one hour if you don't](https://ai.google.dev/gemini-api/docs/generate-content/caching)), and the entry expires when that elapses no matter how much traffic hits it. Reads don't extend it. You can update the TTL or delete the cache, and unlike the other two providers you *must* think about deletion, because the meter runs until expiry. ## Minimums, and why your cache is silently doing nothing Every provider has a floor below which caching simply doesn't happen. Anthropic's varies by model — [512 tokens on Opus 5, 1,024 on Sonnet 5, 4,096 on Haiku 4.5](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) — and short prompts are processed without caching and **no error is returned**. OpenAI requires 1,024 tokens through the breakpoint, and notes that on models before GPT-5.6 the real minimum ranges up to 2,048, so prompts just over 1,024 may cache inconsistently. Gemini's implicit minimum is [2,048 to 4,096 tokens depending on model](https://ai.google.dev/gemini-api/docs/caching). This is the most common cause of "why is my cache read always zero," and it fails quietly by design. If both your write and read counters are zero, check the floor before you check anything else. ## The strategies worth knowing by name **Prefix ordering** — static content first, variable content last. Universal advice, and the highest-leverage thing you can do. > **Common pattern:** putting everything static — instructions, persona, tool definitions — in the system prompt, and everything variable — the actual user request — in the user message, is one of the simplest ways to apply this. It naturally keeps your prefix identical across requests, which is what caching needs, without any extra bookkeeping on your end. **Breakpoint placement** — put the marker on the last block that is *identical* across requests, not on the last block in the prompt. Anthropic documents this failure mode precisely: if your breakpoint sits on a block containing a timestamp or the incoming user message, every request writes a fresh entry and none of them ever read. You pay the write premium forever and never collect. **Pre-warming** — loading a prefix into cache before real traffic arrives, to kill the first-request latency penalty. **Keep-alive pinging** — firing cheap requests to refresh an entry that would otherwise lapse. Viable where refreshes are free; pointless where TTL is wall-clock. **Lifecycle deletion** — explicitly destroying cache resources when a job ends. Only relevant on Google, and only because that's the only place where forgetting costs money. ## What to watch in the response Each provider reports cache activity in its own fields, and you cannot manage what you don't log. - **Anthropic:** `cache_creation_input_tokens` (writes), `cache_read_input_tokens` (reads), and `input_tokens` — which counts only tokens *after* your last breakpoint, not your whole prompt. Writes are further broken out by TTL under `cache_creation`. - **OpenAI:** `cached_tokens` (reads) and, on GPT-5.6 and later, `cache_write_tokens` (writes), in `input_tokens_details` on Responses or `prompt_tokens_details` on Chat Completions. - **Gemini:** `cachedContentTokenCount` on Vertex, `usage.total_cached_tokens` via the Interactions API. Note the asymmetry. Reads and writes are per-request numbers that show up in your logs. Google's storage fee is a property of a cache *resource*, not a request — so it appears on no per-request counter anywhere. It is structurally invisible to request-level observability, which is exactly why it's the charge teams discover late. ## Vocabulary is the whole game None of this is difficult once the words are pinned down. The difficulty is that three providers borrowed each other's terminology without borrowing each other's mechanics, and the resulting near-synonyms hide real money. ## Next: [Prompt caching pricing 101 — how to reduce your bill](/updates/llm-prompt-caching-pricing-guide-and-cost-breakdown) The cross-provider cost model, the side-by-side pricing and operational-constraints charts, and three portable rules of thumb that survive the next repricing. ## Deep dives for different inference providers - **[Anthropic prompt caching: the TTL bet, worked four ways](/updates/llm-prompt-caching-anthropic-how-to-and-pro-tips)** - **[OpenAI prompt caching after GPT-5.6: free writes are over](/updates/llm-prompt-caching-openai-how-to-and-pro-tips)** - **[Gemini context caching: the only cache that bills you for doing nothing](/updates/llm-prompt-caching-gemini-how-to-and-pro-tips)** *Coolhand Labs tracks inference pricing across providers, including the dimensions most catalogs don't model. [See what we track](https://coolhandlabs.com/inference-apis).* --- [← All Updates](https://coolhandlabs.com/updates.md) · [RSS Feed](https://coolhandlabs.com/updates.rss)