Prompt Caching
Prompt caching is an API feature that stores the processed start of a prompt so later requests with the identical start can reuse it, cutting both cost and time to first token. It works on exact prefixes: the shared opening of the prompt must match token for token. The cached portion usually holds stable content such as system instructions, tool definitions, and reference documents. As of November 2026, Anthropic, OpenAI, and Google all offer it, and cached reads on the newest models can cost as little as 2.5% to 5% of the normal input rate. Caching does not change the model's output.
Pay full price for your long prompt once, then reuse it for a fraction of the cost.
Category: Inference & Deployment
How prompt caching works
When a model reads a prompt, it computes internal state for each token. Under the hood that work is held in the KV cache. Without caching, the provider repeats it on every call, even when 95% of your prompt never changes. Prompt caching keeps that work for a limited time and exposes the saving as a billing feature.
A transformer's state at any token depends on every token before it. That is why the cache is prefix-based. If token 10 changes, everything after token 10 must be recomputed, even when the text after it is identical.
The one rule that matters: Put stable content first and changing content last. A timestamp at the top of your system prompt invalidates everything after it on every single request.
Provider comparison (November 2026)
| Feature | Anthropic | OpenAI | Google Gemini |
|---|---|---|---|
| How it turns on | Top-level cache_control, or up to 4 explicit breakpoints | Automatic; GPT-5.6+ also allows up to 4 explicit breakpoints | Implicit caching on by default for Gemini 2.5 and newer |
| Minimum prompt | 512 tokens on Opus 5.5, Sonnet 5.5, Haiku 5.5 | 1,024 tokens on GPT-5.6+ | 4,096 tokens on Gemini 4 Argon and 3.8 Flash |
| Lifetime | 5 minutes (refreshed on use) or 1 hour | 30 minutes or 12 hours on GPT-5.6+ | Not stated on the caching page |
| Read price | 5% of input on Opus and Sonnet 5.5; 2.5% on Fable 5.1 | 5% on GPT-6.1 Sol and Terra; 10% on many others | Listed per model on the pricing page |
| Write price | 1.25× input (5 minutes), 2× input (1 hour) | 1.25× input (30 minutes), 2.5× (12 hours) on GPT-5.6+; none on earlier models | Not stated on the caching page |
| Usage field | cache_read_input_tokens | usage.input_tokens_details.cached_tokens | usage.total_cached_tokens |
These rules change often: Minimums, lifetimes, and prices vary by model and have changed several times. On older models the numbers differ, and Gemini's explicit caching is not supported on Gemini 4 Argon and later. Check the current docs before you budget.
Worked example: when caching saves money
Take a 50,000-token stable prefix on Claude Opus 5.5: input at $4 per million tokens, 5-minute writes at $5, 1-hour writes at $8, and reads at $0.20. These figures count the prefix only, not new user text or output.
| Scenario | No cache | With cache | Result |
|---|---|---|---|
| 10 calls, one minute apart, 5-minute cache | $2.00 | $0.25 write + 9 × $0.01 reads = $0.34 | Saves 83% |
| 2 calls, one minute apart, 5-minute cache | $0.40 | $0.25 + $0.01 = $0.26 | Saves 35%; one read pays back the write |
| 6 calls, 10 minutes apart, 5-minute cache | $1.20 | 6 × $0.25 writes = $1.50 | Costs 25% more, because every call misses |
| 6 calls, 10 minutes apart, 1-hour cache | $1.20 | $0.40 write + 5 × $0.01 reads = $0.45 | Saves 62.5% |
The third row is the trap. Caching a prompt that expires before the next call means paying the write premium again and again. Match the lifetime to your real traffic, and measure the hit rate.
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-opus-5-5",
max_tokens=1024,
system=[{
"type": "text",
"text": LONG_STABLE_INSTRUCTIONS,
"cache_control": {"type": "ephemeral"}, # 5-minute default
}],
messages=[{"role": "user", "content": "Summarize section 3."}],
)
usage = response.usage
print("written to cache:", usage.cache_creation_input_tokens)
print("read from cache: ", usage.cache_read_input_tokens)
print("not cached: ", usage.input_tokens)What breaks a cache
Anthropic's docs describe a hierarchy of tools, then system, then messages. A change at one level invalidates that level and everything after it.
| Change | Effect on Anthropic caching | How to avoid it |
|---|---|---|
| Editing tool definitions | Invalidates everything | Keep tools stable and in a fixed order |
| Editing the system prompt | Invalidates system and messages | Move dates and user details later in the prompt |
| Changing tool_choice or thinking settings | Invalidates the message cache | Keep request settings consistent |
| Adding or removing images | Invalidates the message cache | Keep stable images early in the prompt |
| Waiting past the lifetime | Cache expires | Use a longer lifetime for slower traffic |
OpenAI's guidance has the same core logic, since the full rendered prefix must match. Its caches also live on individual machines, so very high request rates can overflow to machines without your cached prefix. OpenAI suggests about 15 requests per minute per prompt_cache_key on older models.
Prompt caching vs related ideas
| Technique | What is reused | Needs an exact match? | Output changes? |
|---|---|---|---|
| Prompt caching | Processed prompt prefix; the model still runs | Yes, on the prefix | No, it still generates fresh output |
| KV cache | Keys and values during one generation | Internal mechanism | No |
| Semantic or response caching | A saved final answer, with no model call | Often similarity-based | Yes, it returns the old answer, which can be wrong for a new question |
Honest caveats
- Cached tokens still fill the context window. Anthropic's docs are explicit that caching changes what you pay for those tokens, not whether they count.
- A cache hit is not guaranteed. OpenAI notes that even a
prompt_cache_keydoes not guarantee a hit. Build budgets from your measured hit rate. - Short prompts get nothing. Prompts under the minimum are processed normally, and Anthropic does not return an error.
- Output tokens are unaffected. If most of your bill is output, caching helps less than you might expect.
- Isolation differs by platform. Anthropic isolates caches per workspace on its own API and per organization on some cloud platforms, and OpenAI does not share caches across organizations. Review the rules if your data is sensitive.
Frequently asked questions
Does prompt caching change the answer? No. The model still generates a new response. Only the prompt-processing work is reused.
How do I know it is working? Read the usage fields in the API response. Cached token counts should rise on the second matching request.
Is caching automatic? On OpenAI and on Gemini's implicit caching, yes for supported models. On Anthropic you switch it on with cache_control.
Why is my hit rate zero? Usually the start of the prompt changes on each call, the prompt is under the minimum, or the cache expires between requests. Check for timestamps, reordered tools, and unstable JSON key order.
Does it work with long conversations? Yes. As a chat grows, each new turn extends the prefix, so earlier turns can be read from cache while only the new part is processed. Anthropic's automatic mode moves the breakpoint forward for this case.
LumiChats gives you dozens of models in one place, and prices and caching rules differ widely between them. If you build your own app on these models, stable prompt prefixes are one of the simplest ways to cut your API bill.