When Prompt Caching Actually Breaks Even: We Mapped the Latency vs Cost ROI

Break-even curves for prompt caching across providers and cache windows

Every prompt caching post claims 90% cost savings. Portkey's analysis shows break-even at 4.3 reads for Sonnet under realistic conditions—not 1.4. HCODX's calculator puts the 1-hour cache break-even at 12 reads for conservative pricing assumptions. The arXiv empirical study measured 13–31% TTFT improvement in production, not the "up to 85%" vendors advertise. The gap between marketing and practice is the actual story.

Vendors quote headlines without workload context. Engineers implement caching and hit either disappointing ROI or—worse—increased costs from naive cache layout. We mapped the latency vs cost ROI across providers, workload shapes, and real hit-rate distributions to show exactly when caching wins, when it's marginal, and when it costs you more.

TL;DR

• Prompt caching cuts cached-token costs 50–90% across providers, but overall per-request savings are 41–80% because you still pay full price for output tokens (arXiv, "An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks," 2026)
• Break-even hit count varies by 8× across published analyses: 1.4 reads per write (5-min TTL, stable prefix) to 12 reads (1-hr TTL, conservative HCODX calculator)—the spread is the signal
• Realistic TTFT improvement is 13–31% in production, not the "up to 85%" benchmark claim; a 1,024-token prompt sees only ~7% TTFT gain (Redis benchmark analysis, 2026)
• Three failure modes can make caching net-negative: prefix fragmentation, cold prefixes, and retry storms
• The go/no-go threshold: cacheable prefix ≥1,024 tokens, ≥30% of total input stable, ≥1 req/min for that template (Tian Pan, "What Real Hit Rates Look Like & When Prompt Caching Actually Breaks Even," 2026; Portkey, "The Hidden Costs of AI," 2026)

The Headline Numbers Don't Transfer to Your Workload

Anthropic advertises 90% cost reduction on cached tokens and 85% latency improvement for long prompts. Both numbers are technically true—for fully cached, static prefixes on isolated benchmarks. In production, the picture is different.

A cross-provider empirical study measured caching across four models on long-horizon agentic workloads. Results: API costs fell by 41–80% and time-to-first-token improved by 13–31% (arXiv, "An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks," 2026). Not 90% and 85%—and agentic tasks are among the most cache-friendly workloads.

The gap has two causes.

Cached token price ≠ per-request cost. Anthropic prices Claude Sonnet at $3.00/MTok for uncached input and $0.30/MTok for cached input (ByteIota, "Anthropic Prompt Caching Cuts AI API Costs 90%," 2026). That's a 90% discount on those specific tokens. But you still pay full price for the uncached portion of your input and all output tokens. If your cached prefix is 50% of total input, your effective overall savings is closer to 45%—not 90%.

TTFT gains ≠ end-to-end latency gains. Caching shortens the time until the model starts generating. But streaming overhead, output generation, tool-call round-trips, and retries add latency that caching doesn't touch. If an agent spends 4,000ms on output, a 200ms TTFT improvement is nearly invisible to the end user.

Cached-token discount vs per-request cost reduction: the gap that vendor benchmarks don't show

Assume 40–70% overall cost reduction and 15–30% latency improvement in production. Treat 90%/85% as ceiling values you approach only with very long, very stable prefixes and high hit rates.

Break-Even Depends on Hit Rate, Provider, and Prefix Design

The standard break-even math is elegant. Cache writes cost 1.25× the normal input token price; cache reads cost 0.1× (Caylent, "Amazon Bedrock Prompt Caching: Saving Time and Money in LLM Applications," 2026). Per prefix token:

• Extra cost to write to cache: +0.25×
• Saving per cache read vs no caching: 0.9×
• Break-even: 0.25 / 0.9 = ~1.4 reads including the write

That's where the "1.4 reads" figure comes from (Zylos AI, "Prompt Caching & KV-Cache Optimization for Long-Running AI Agents," 2026; Tian Pan, 2026). It's mathematically correct for a perfectly stable prefix within a 5-minute window. The range of published numbers tells a different story.

The range—1.4 to 12—isn't a data disagreement. It reflects what happens when you change the assumption about prefix stability. The simple 1.4 formula assumes every request hits the same cache entry. HCODX's 12-read figure accounts for more conservative usage patterns where the effective cache hit rate is much lower.

The other gap: prefix size fraction matters. Portkey's 4.3-read scenario had a smaller cacheable prefix relative to total input (Portkey, "The Hidden Costs of AI," 2026). A 10,000-token prompt breaks even at just 2 cache hits; prompts under 1,024 tokens aren't eligible for Anthropic prefix caching at all.

For OpenAI, automatic caching gives a ~50% discount on cached input tokens (Introl, "Prompt Caching Infrastructure," 2025). With a 1.25× write and 0.5× read:

• Extra write cost: 0.25×
• Saving per read: 0.5×
• Break-even: 0.25 / 0.5 = ~1.5 reads

Near-identical break-even, but much lower savings per read above break-even.

The missing variable in every simple formula: prefix stability. Any dynamic content before the cache boundary—a timestamp, user ID, trace ID, or per-request instruction—creates a new cache entry that won't be reused. Your effective hit rate collapses, and break-even moves from the 1.4 end of the spectrum toward the 12-reads end.

The heatmap makes visible what the simple formula hides: prefix stability is the dominant variable. A workload with 25% prefix stability at a 1-hour TTL needs 8–12 reads just to break even. That's not "enable caching and win"—that's "fix your prompt structure first or don't bother."

Latency Isn't Free: When Caching Actually Costs You Speed

Caching is marketed as 85% faster. That's TTFT on isolated benchmarks. Production is closer to 13–31%—and only when caching is actually working.

The Redis benchmark is blunt: a 1,024-token prompt saw only ~7% TTFT improvement; 100,000+ token prompts saw 67–79% (Redis benchmark analysis, 2026). Latency gains scale directly with prefix length. If your cached prefix is short, the improvement is negligible.

ByteIota documented the cherry-picked best case: a 100,000-token RAG workload where average response time dropped from 3.2 seconds to 0.5 seconds—an 85% reduction (ByteIota, "Anthropic Prompt Caching Cuts AI API Costs 90%," 2026). That's the vendor scenario. It's valid. It doesn't generalize.

Bursty traffic flips the math. If your traffic is sparse—quiet periods followed by sudden spikes—you're constantly paying write costs for cold cache entries. Every write is 1.25× and full TTFT. If traffic never sustains long enough for reads to accumulate within the TTL window, caching increases your average cost per request rather than reducing it.

Retry amplification. A failed tool call or a 429 error can trigger retries with modified context. Each retry with changed context writes a new cache entry—and misses the old one. At a 10% retry rate, you're generating 10% more cache writes than expected, with no corresponding reads. The cache is working; it's just working against you.

The practical rule: caching helps latency only for large, stable prefixes and consistent traffic above ~1 request/minute per template. Below that threshold, the cold-start penalty can outweigh the benefit entirely.

When Caching Hurts: Three Failure Modes to Watch

Naive caching can increase cost and latency. Zylos AI states this directly: naive strategies "paradoxically increase latency and cost by creating cache entries that are never reused" (Zylos AI, "Prompt Caching & KV-Cache Optimization for Long-Running AI Agents," 2026). Their framing: caching is "a correctness problem masquerading as a performance problem."

Failure mode 1: Prefix fragmentation. The most common. Dynamic content appears early in the prompt—a timestamp, user ID, session token, current date, or any per-request variable. Each variant creates a unique cache entry never seen again. Write costs accumulate; read savings never materialize. The fix is structural: put all static content first, push every dynamic element after the cached boundary. "Static content first, dynamic content last" (ngrok, "Prompt caching: 10x cheaper LLM tokens, but how?," 2026).

Failure mode 2: Cold prefixes. You cache a template used once per week. The write cost from the first call isn't amortized before the TTL expires. Portkey's threshold: at least 1 request per minute for a given template to justify caching (Portkey, "The Hidden Costs of AI: Understanding Prompt Caching and When to Use It," 2026). Tian Pan adds: expected requests per 1-hour cache window should exceed 3. Below those thresholds, you're paying for cache infrastructure you never actually use.

Failure mode 3: Retry storms. An agent loop with a 10% retry rate creates roughly 10% more cache writes than planned. If retried prompts carry new context—different tool results, revised instructions—those writes also miss on subsequent reads. Your effective write rate exceeds your read rate. Fix: measure cache hit rate per prefix, not as a global average. A prefix with hit rate below 1.0 (more writes than reads) is actively losing you money every call.

Good structure (static-first) vs bad structure (dynamic content in prefix): the difference between 70% hit rate and 5%

The Real ROI Depends on Your Workload Shape

Most posts show one scenario. Real teams have 5–10 prompt templates with different hit rates, prefix sizes, and traffic patterns. Global ROI averages obscure the distribution—and the tail is where caching actually fails.

TokenMix production data: teams implementing caching reduce input token costs by 60–85% on average (TokenMix, "Prompt Caching Guide 2026: Cut AI API Costs 50–95% Proven," 2026). The fine print: that requires cache hit rates above 60% and cacheable prefixes above 80% of total input. Most workloads don't hit both conditions simultaneously.

The Introl finding is sobering: only ~31% of LLM queries exhibit strong semantic similarity to prior ones (Introl, "Prompt Caching Infrastructure: Reducing LLM Costs and Latency," 2025). For structured workloads with stable templates—RAG, agents with fixed system prompts—you can beat that. For conversational or exploratory queries, you typically can't.

Workload shapes where caching tends to pay off:

• RAG pipelines with large shared document context (input cost reduction often 60–80%; prefix is large, stable, and reused heavily)
• Contact-center and classification templates with high request volume over few prompt variants (hit rates can exceed 60% with proper prefix design)
• Long-running agents with static system prompts and tool definitions (savings vary widely: 40–70%, depending on retry rate and prefix fragmentation)

Workload shapes where caching often doesn't pencil out:

• Interactive chat with per-user system prompts (personalization fragments the prefix, collapsing hit rates)
• Low-traffic templates with under 1 request/minute per template
• Exploration and research workflows where each prompt is unique by design

The chart is an estimate, not a single-study measurement. The actual distribution within your workload depends on prefix design, traffic volume, and how much personalization you've baked into the cached portion. Treat it as a starting hypothesis, not a guarantee.

How to Measure If Caching Is Worth It for Your Workload

Three checks. If all three pass, pilot it. If any fail, fix the root cause first.

Check 1: Is the cacheable prefix ≥1,024 tokens AND ≥30% of total input? Anthropic prefix caching doesn't support prompts under 1,024 tokens (Portkey, 2026; Tian Pan, 2026). Below that, there's nothing to cache. The ≥30% threshold ensures the discounted portion is large enough to matter—if only 10% of your tokens are cached, the effective savings on total cost is minimal even at 90% per-token discount.

Check 2: Is expected traffic ≥1 request per minute for that template? Portkey's guidance is explicit: below that rate, engineering complexity outweighs ROI (Portkey, 2026). Tian Pan adds: conservative break-even requires more than 3 requests per 1-hour cache window. If your template gets 10 requests per day, skip caching.

Check 3: Is the prefix truly stable? Log 100 requests. If the first 1,024+ tokens differ across more than 30% of requests, prefix fragmentation will erode your hit rate. Fix the structure first: move timestamps, user IDs, and any request-specific content after the cache boundary.

If all three pass: enable caching for 10% of traffic. After 100+ requests, check hit rate. Above 40%: likely profitable. Below 20%: net-negative—diagnose the fragmentation before scaling.

Monthly ROI sanity check: (uncached monthly spend on this template) − (cached monthly spend + engineering time × hourly rate). If the delta is positive within 6 months, commit fully.

Provider Comparison: Anthropic vs OpenAI vs Bedrock

There's no universal winner. Provider choice depends on workload shape and how much engineering complexity you can absorb.

Anthropic: Explicit prefix caching. 1.25× write premium, 0.1× read cost (90% discount). TTL of 5 minutes to 1 hour. 5 million token cache limit (ByteIota, 2026). Best for: RAG with large shared document context, agent systems with long stable system prompts, workloads where maximizing per-read savings justifies the configuration overhead.

OpenAI: Automatic caching, no code changes required. ~50% discount on cached input tokens (Introl, "Prompt Caching Infrastructure," 2025; TokenMix, 2026). Lower per-token savings than Anthropic, but near-zero engineering investment. Best for: interactive chat, teams that want passive caching benefits without prompt refactoring.

Amazon Bedrock: Explicit caching, 1.25× write, 0.1× read, no separate storage fees for the cached context (Caylent, "Amazon Bedrock Prompt Caching: Saving Time and Money in LLM Applications," 2026). Economics nearly identical to Anthropic. Note: Google's context caching includes per-hour storage charges, which raises break-even to roughly 3–4 queries per hour for large contexts (TokenMix, 2026).

Break-even math is "essentially identical" across Anthropic and Bedrock at around 1.4 reads for the 5-minute case (Tian Pan, 2026). Anthropic's higher per-read discount (90% vs 50%) means better ROI above break-even for high-traffic workloads. OpenAI's automatic approach saves engineering time—important when prompt refactoring cost is non-trivial.

If you want to run the same prompt on all three with caching enabled and measure actual cost and latency numbers, Mesh API's compare endpoint does exactly that—one API call, results from every major provider.

The Path Forward

Start narrow. Pick the 2–3 templates that account for 80% of your token spend. Audit each for prefix stability: no dynamic content before the cache boundary, prefix ≥1,024 tokens, traffic ≥1 request/minute. Enable caching for those templates only. Measure hit rate after 100+ requests.

If hit rate is above 40% and prefixes are long, caching will likely pay off. If hit rate is below 20%, the prefix is fragmented or traffic is too low. Fix the root cause before scaling.

If caching doesn't work for your workload shape, the alternatives—model routing to cheaper models for lower-complexity tasks, batching, quantization—often deliver comparable savings without the prompt engineering overhead. DigitalApplied frames prompt caching as "the highest-ROI first move" in the LLM FinOps toolkit (DigitalApplied, "AI Inference Cost Optimization: FinOps Playbook 2026"), but that's conditional on your workload qualifying.

The honest baseline: vendor 90%/85% are ceiling values. Production reality is 40–70% cost reduction and 13–31% latency improvement. The gap closes with long, stable prefixes and consistent traffic. It widens—or goes negative—without them.

FAQ

Does prompt caching actually reduce latency, or just cost? Both, but not equally. TTFT improves 13–31% in realistic production workloads (arXiv, 2026), not the 80–85% vendor marketing figure. A 1,024-token prompt sees only ~7% TTFT improvement; 100,000+ token prompts see 67–79% (Redis benchmark analysis, 2026). Latency gains scale with prefix length. Output generation, retries, and tool-call latency aren't affected by caching and often dominate end-to-end time.

What's the actual break-even hit count? For Anthropic or Bedrock (90% read discount, 1.25× write): ~1.4 reads at 5-minute TTL, 2–3 reads at 1-hour TTL under standard assumptions (Zylos AI, 2026; Tian Pan, 2026). But that assumes a stable prefix. Any dynamic content in the prefix fragments it, raising break-even to 5–10+ reads. The HCODX calculator puts the conservative 1-hour break-even at 12 reads. Measure hit rate per prefix—not globally.

Can prompt caching actually hurt performance or cost? Yes. Fragmented prefixes create writes with no reads, making caching net-negative. Infrequent traffic means write costs aren't amortized. Retry storms create new cache entries faster than they're reused. Zylos AI explicitly documents that naive caching "paradoxically increases latency and cost" (Zylos AI, 2026). The fix: audit prompt structure, measure hit rate per prefix, and skip caching for templates below the go/no-go thresholds.

Which provider has the best caching ROI? Depends on workload. Anthropic's 90% read discount provides the highest per-read savings for large reusable prefixes—better for high-traffic RAG and agents. OpenAI's automatic caching requires no code changes—better when prompt refactoring cost is high relative to expected savings. Break-even reads are nearly identical across providers at ~1.4 for the 5-minute case (Tian Pan, 2026). What differs is savings rate above break-even.

How do I know if my workload qualifies? Three checks: (1) cacheable prefix ≥1,024 tokens AND ≥30% of total input? (2) Expected traffic ≥1 req/min for that template? (3) Prefix truly stable? If yes to all three, pilot on 10% of traffic and measure hit rate over 100+ requests. Above 40%: likely profitable. Below 20%: fix prefix stability first. Below the thresholds: skip caching and pursue other optimizations (Tian Pan, 2026; Portkey, 2026).