Prompt Caching for AI Agents: Cut Latency and Cost Guide

Prompt caching is one of the few AI agent optimizations that improves both cost and latency simultaneously with no tradeoff — but only if the prompt is structured so the cache actually hits. Here's how to set it up correctly.

Step 1: Understand What Actually Gets Cached

Caching stores a processed representation of a prompt prefix — everything up to a designated breakpoint. A later call with the identical prefix skips reprocessing those tokens.

python

The cache_control marker tells the API where the cacheable prefix ends. Everything before it needs to be byte-for-byte identical across calls for the cache to hit.

Step 2: Structure Prompts With Static Content First

python

Any value that changes between calls — a timestamp, a request ID, dynamic user context — invalidates the cache for that entire request if it sits before the breakpoint. Moving dynamic content to the end of the prompt, after everything cacheable, is what keeps the cache hit rate high.

Step 3: Cache Tool Definitions and Large Reference Documents Too

python

Tool definitions are often static across an entire agent session and can be sizable (many tools with detailed schemas) — caching them separately from the system prompt captures the same savings for content that's just as repetitive.

Step 4: Account for Cache TTL in Agent Loops

python

Default cache TTLs are short (commonly around 5 minutes of inactivity). If an agent's loop includes a slow tool call — a long-running search, a large file operation — the gap between one model call and the next can exceed the TTL, silently reverting to full-price, full-latency processing on the next turn.

Step 5: Measure Cache Hit Rate, Not Just Assume It's Working

python

A prompt restructure that looks correct in code can still fail to cache in practice — for example, a tool list assembled in a different order on each call (from a set or dict without stable ordering) produces a different byte sequence every time even though the content is logically the same. Measuring the actual hit rate is the only way to confirm caching is working as intended.

Step 6: Know When Caching Doesn't Help

python

Caching pays off in proportion to how much static content is repeated across how many calls. A one-shot request with a short system prompt has little to gain; a multi-turn agent loop reusing a large system prompt and tool catalog across dozens of calls is exactly the shape caching is built for.

Key Takeaways

Prompt caching only helps when the cached prefix is byte-for-byte identical across calls, which means static content (system prompt, tool definitions) needs to come before any dynamic, per-request content in the prompt structure, cache TTLs are short enough that slow tool calls in an agent loop can silently cause cache misses, and measuring actual cache hit rate — not just assuming a correct-looking implementation is working — is the only way to confirm the savings are real.

Frequently Asked Questions

What exactly gets cached in prompt caching?

A prefix of the prompt — typically the system instructions, tool definitions, and any static reference content placed before the dynamic, per-call content. The API stores a processed representation of that prefix so a subsequent call with an identical prefix can reuse it instead of reprocessing the same tokens from scratch.

Why did my prompt caching stop working after I added a timestamp to the system prompt?

Cache matching requires an exact match on the cached prefix. A timestamp, request ID, or any other value that changes on every call, if placed before the cache breakpoint, changes the prefix on every request and invalidates the cache every single time. Dynamic content needs to go after the cached portion of the prompt, not inside it.

How long does a prompt cache last?

Cache TTLs are typically short by default — around 5 minutes of inactivity for Anthropic's ephemeral cache, with an extended-duration option available for longer gaps between calls. For an agent loop where a tool call takes longer than the TTL to return, the cache can expire between turns, so it's worth measuring actual gaps between calls in a long-running agent.

Is prompt caching worth it for a simple, one-shot LLM call?

Usually not — caching pays off when the same prefix is reused across multiple calls, which is the normal shape of an agent loop (many turns sharing the same system prompt and tool definitions) but not a single isolated request. A one-shot call with a short prompt has little static content to cache and won't benefit meaningfully.