Skip to content
AI & Tools

Prompt Caching Explained: How Anthropic, OpenAI and Google Cache Prompts

AIDToolStack EditorialNovember 7, 20269 min read
Prompt CachingLLM APIsCost OptimizationClaudeOpenAIGemini
Share:Twitter

Look at what an agent loop actually sends. Every turn carries the same tool definitions, the same long system prompt, the same reference docs and the whole conversation so far, plus one new message at the end. If that's 20,000 tokens of repeated context, you pay to process those 20,000 tokens again on every call, and you wait for them too.

Prompt caching is the providers' answer. The model's processed state for a prompt prefix is kept for a while (OpenAI's caching guide says it stores KV tensors, not the tokens), and the next request that starts with exactly the same prefix reuses it at a discount. All three big providers do this. They differ in how you turn it on, how long it lasts, what writes cost, and how short a prompt can be before caching silently doesn't happen. This post goes through each as of November 2026.

The one rule: it's a prefix match

Caching works on the start of the prompt, and it needs an exact match. Anthropic's prompt caching docs spell out the order the prefix is built in: tools, then system, then messages. OpenAI routes requests by a hash of the initial tokens, tool definitions included. Change one token early in the prompt and everything after it is a miss.

Three requests laid out left to right: tools, system prompt, reference docs and history form a cached prefix ending at a breakpoint; request 2 adds a new question after the same prefix and hits the cache; request 3 changes a timestamp inside the system prompt, so the prefix no longer matches and the whole request misses
Three requests laid out left to right: tools, system prompt, reference docs and history form a cached prefix ending at a breakpoint; request 2 adds a new question after the same prefix and hits the cache; request 3 changes a timestamp inside the system prompt, so the prefix no longer matches and the whole request misses

So the design rule is the same everywhere. Stable content goes first, things that change per request go last. The classic mistake is a system prompt that begins "Current time: 14:32:07". That one line at the top makes every request a cache miss. Anthropic's docs say exactly this: content that changes every request, like timestamps, belongs after the last cache breakpoint.

Caching doesn't change what the model generates. Both Anthropic and OpenAI say so. It's purely a cost and latency feature.

Anthropic: you mark what to cache

Claude's API caches only what you tell it to. There are two ways:

  • Automatic caching: put one cache_control field at the top level of the request. The breakpoint sits on the last cacheable block and moves forward as the conversation grows. The pricing page calls this the recommended starting point.
  • Explicit breakpoints: put cache_control on individual content blocks, up to 4 per request.
import anthropic

client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": STYLE_GUIDE_AND_EXAMPLES,   # long and identical on every call
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": question}],
)

u = resp.usage
print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)

The usage fields tell you what happened. cache_creation_input_tokens were written to the cache, cache_read_input_tokens were served from it, and input_tokens is whatever came after the last breakpoint. If both cache fields are 0, nothing was cached, and the docs say the usual reason is a prompt below the minimum length. There's no error, so log these fields from day one.

The details that matter:

  • Lifetime. The default is 5 minutes, refreshed free each time the cache is read. Add "ttl": "1h" to cache_control for an hour. If you mix them, 1-hour entries have to come before 5-minute ones.
  • Minimum length. 512 tokens on the current generation (Opus 5.5, Sonnet 5.5, Haiku 5.5 and others), 1,024 on models such as Sonnet 4.6, and up to 4,096 on Opus 4.5, Opus 4.6 and Haiku 4.5.
  • Price. A 5-minute write costs 1.25x the base input price and a 1-hour write 2x. Reads cost 0.1x on most models, 0.05x on Opus 5.5 and Sonnet 5.5, and 0.025x on Fable 5.1 and Mythos 5.1. These stack with the Batch API discount.
  • Lookback. On a read, the API checks at most 20 positions back from each breakpoint. In a long agent loop, move the breakpoint forward (automatic caching does this for you) or you'll fall off the end.
  • Isolation. On the Claude API, caches are per workspace. Organizations never share them.

OpenAI: on by default, and newly priced

OpenAI caches automatically on supported models, with no code changes. The caching guide now splits behaviour by model generation, which is the thing most older blog posts get wrong.

GPT-5.6 and later (for example gpt-6.1-sol):

  • The minimum is 1,024 tokens of visible input.
  • Caching is implicit by default (a breakpoint at the end of the latest eligible message), or explicit with prompt_cache_options and per-block prompt_cache_breakpoint, up to four writes per request.
  • ttl is "30m" (default) or "12h", the latter on the Responses API only.
  • Writes cost 1.25x input for 30 minutes and 2.5x for 12 hours. Reads cost 0.1x, or 0.05x on GPT-6.1 Sol and Terra. On the pricing page, gpt-6.1-sol is $2 input, $0.10 cached, $2.50 for a 30-minute write.
{
  "model": "gpt-6.1-sol",
  "prompt_cache_options": { "mode": "explicit", "ttl": "30m" },
  "prompt_cache_key": "support-bot-v3:tenant-42",
  "input": [
    {
      "role": "developer",
      "content": [{
        "type": "input_text",
        "text": "<long, stable instructions>",
        "prompt_cache_breakpoint": { "mode": "explicit", "ttl": "12h" }
      }]
    },
    { "role": "user", "content": [{ "type": "input_text", "text": "<this turn>" }] }
  ]
}

One catch from the guide: the top-level instructions field can't hold a breakpoint, so reusable instructions go in a developer message, as above.

Earlier models (GPT-5.5 and before) work the old way. There's no cache-write charge, the minimum varies with request settings, and retention is prompt_cache_retention: "in_memory" (roughly 5 to 10 minutes idle, up to an hour) or "24h". On these models prompt_cache_key matters for hit rates. Caches live on individual machines, and the guide says traffic above about 15 requests per minute for one prefix and key can overflow to other machines and miss. Split busy traffic across keys.

Read the result from usage.input_tokens_details.cached_tokens, with cache_write_tokens beside it on newer models. Cached tokens still count toward your tokens-per-minute rate limit, and there's no way to clear the cache manually.

Google: implicit by default, explicit on older models

Gemini has two mechanisms, and which you get depends on the model.

Implicit caching is on by default for Gemini 2.5 and newer, per Google's caching docs. The minimum is 4,096 tokens on current models (Gemini 4 Argon, 3.8 Flash, 3.6 Flash, 3.1 Pro Preview) and 2,048 on Gemini 2.5. Google passes savings on when a request hits but doesn't guarantee them. There's also a Gemini 4 rule for agent loops: the full previous request is only cached if the new request adds at least 2,050 tokens.

Explicit caching is where you create a named cache object and reference it. It's beta, it works through generateContent on Gemini 3 and 2.5 models, and it isn't supported on Gemini 4 Argon or later. The default TTL is one hour.

from google import genai
from google.genai import types

client = genai.Client()

cache = client.caches.create(
    model="gemini-3.8-flash",
    config=types.CreateCachedContentConfig(
        display_name="product-manual",
        system_instruction="Answer only from the manual.",
        contents=[MANUAL_TEXT],
        ttl="3600s",
    ),
)

resp = client.models.generate_content(
    model="gemini-3.8-flash",
    contents="How do I reset the device?",
    config=types.GenerateContentConfig(cached_content=cache.name),
)

Explicit caches are billed for storage by the hour on top of the cheaper cached-token price. On the pricing page, Gemini 3.8 Flash is $0.75 input, $0.075 cached and $0.50 per million tokens per hour of storage, all doubling on 1 January 2027. Gemini 3.1 Pro Preview's storage is $4.50 per million per hour. An idle explicit cache costs money, so give it a TTL that matches your traffic.

Side by side

AnthropicOpenAI (GPT-5.6+)Google
Turned on bycache_control (top level or per block)Default; explicit mode optionalImplicit default; explicit via caches.create
Lifetimes5 min (refreshed on read), 1 h30 min, 12 hImplicit: unstated; explicit: 1 h default, you set TTL
Minimum prefix512 tokens on current models1,024 tokens4,096 tokens on current models
Write cost1.25x (5 min), 2x (1 h)1.25x (30 min), 2.5x (12 h)Storage per million tokens per hour (explicit)
Read cost0.1x, 0.05x or 0.025x by model0.1x, 0.05x on 6.1 Sol/TerraModel's cached-token price
Where to lookcache_read_input_tokenscached_tokensusage_metadata

Does it pay off?

Anthropic's pricing page puts it simply: a 5-minute write pays for itself after one read, a 1-hour write after two. Here's a concrete case with a 20,000-token prefix on Claude Sonnet 5.5 ($2 per million input, $2.50 for a 5-minute write, $0.10 for a read):

  • Uncached, the prefix costs $0.04 every call.
  • Cached, the first call costs $0.05 (the write) and each read after it $0.002.
Bar chart of cumulative prefix cost for a 20,000-token prefix on Claude Sonnet 5.5: after 1 call uncached is $0.04 and cached $0.05; after 2 calls $0.08 versus $0.052; after 5 calls $0.20 versus $0.058; after 10 calls $0.40 versus $0.068
Bar chart of cumulative prefix cost for a 20,000-token prefix on Claude Sonnet 5.5: after 1 call uncached is $0.04 and cached $0.05; after 2 calls $0.08 versus $0.052; after 5 calls $0.20 versus $0.058; after 10 calls $0.40 versus $0.068

Ten calls inside the cache window cost $0.40 uncached and $0.068 cached. gpt-6.1-sol happens to have identical input, write and read prices, so the same numbers hold there. The output tokens are unaffected either way, and they're often the bigger part of the bill, as the LLM API pricing comparison shows. Our LLM pricing calculator helps with the rest of the arithmetic.

Where caching loses: one-off requests, traffic so sparse that the cache always expires before the next call, and 1-hour or 12-hour writes for prefixes that only get read once.

Things that quietly break the cache

  • Volatile content near the top. Timestamps, request IDs, the user's name in the system prompt. Move them to the end.
  • Tool lists that change order. Tools come first in the prefix, so any reshuffle invalidates everything. Anthropic's docs say changing tool definitions invalidates the whole cache. The MCP tools spec asks servers to return tools in a deterministic order for this very reason. If your host merges several MCP servers, sort the combined list.
  • Switching tool_choice. On Anthropic it invalidates the message cache. On OpenAI, allowed_tools lets you restrict which tools can be called without changing the tool list, which the function calling guide says helps preserve caching.
  • Toggling features mid-conversation. On Anthropic, adding or removing images, changing thinking settings or effort, or turning web search or citations on or off all invalidate some or all of the cache. Changing the structured output format does too, as the structured outputs post notes.
  • Non-deterministic serialization. If you build the prompt from a dict or JSON object whose key order varies, the bytes differ and so does the prefix.
  • Being under the minimum. A 900-token system prompt never caches on a model with a 1,024-token minimum, and nobody tells you. Our token counter is a quick way to check where you are.

Checklist

  1. Order every prompt as tools, system, long reference material, history, new turn. Nothing volatile before the last breakpoint.
  2. Turn caching on explicitly where you have to (Anthropic, Gemini explicit) and pick the shortest lifetime your traffic keeps warm.
  3. Log the cache fields on every response and alert when the hit rate drops after a deploy.
  4. Keep tool definitions byte-for-byte stable and sorted.
  5. On older OpenAI models, set a prompt_cache_key and spread hot prefixes across keys.
  6. Recheck minimum lengths and multipliers when you change models. They differ per model, even within one provider.
AE

Written by AIDToolStack Editorial

Researched from official documentation, specifications and vendor pages, listed under Sources below. Last checked November 2026.

Sources

  1. developers.openai.com/api/docs/guides/prompt-caching
  2. platform.claude.com/docs/en/build-with-claude/prompt-caching
  3. platform.claude.com/docs/en/about-claude/pricing
  4. developers.openai.com/api/docs/pricing
  5. aistudio.google.com/docs/caching
  6. aistudio.google.com/docs/pricing
  7. modelcontextprotocol.io/specification/latest/server/tools
  8. developers.openai.com/api/docs/guides/function-calling

Related Posts

Explore on AIDToolStack

Discover tools and servers featured in our directory.

MCP Servers

Trigger.dev Jobs MCP ServerManage Trigger.dev background jobs from your AI coding assistant. Monitor job runs, trigger tasks manually, inspect execution logs, manage schedules, and debug failed runs. The server provides operational visibility into your background job infrastructure — see which tasks are running, queued, or failed, and take action. Your AI can help diagnose job failures from execution logs, retry failed tasks, and optimize concurrency configurations for better throughput.Tailscale Network MCP ServerManage your Tailscale mesh VPN network from your AI coding assistant. View connected devices, manage ACLs, configure exit nodes, check device status, and manage DNS settings. The server provides visibility into your Tailscale network — see which machines are online, troubleshoot connectivity issues, manage network access policies, and configure routes. Useful for developers managing distributed infrastructure where Tailscale provides secure networking between services.Portainer Docker MCP ServerManage Docker environments through Portainer's API from your AI coding assistant. Control containers, images, networks, and volumes across multiple Docker hosts and Kubernetes clusters. The server provides tools for container lifecycle management, image pulling and building, stack deployment, and resource monitoring. Portainer's multi-environment management makes this server particularly valuable for teams managing containers across development, staging, and production environments.

Reviews

Sign in to leave a review.

No reviews yet. Be the first!

Comments

Sign in to leave a comment.

No comments yet. Be the first!

Sponsor Our Newsletter

Reach thousands of developers who are actively evaluating AI tools, MCP servers, and dev infrastructure. Our weekly newsletter goes to engaged technical decision-makers.

Growing subscriber baseHigh open rates

All sponsored content is clearly labeled per our editorial policy.