The math behind Anthropic's 5-minute and 1-hour prompt cache options
bigaiguy · x · 2026-11-10
Why does Anthropic only offer 5-minute and 1-hour prompt cache TTLs? It comes down to storage: caches in GPU memory (fast, expensive) break even against re-reading your prompt in 4.5 minutes, while CPU memory (DRAM) breaks even at 1.6 hours.
For a 100K-token Llama 3.3 70B cache, re-reading costs $0.05 (≈55s of an H100 at 50% util, $3.50/hr). Holding cost per GB-hour: GPU $3.50/80GB, CPU $0.0021, SSD $0.000062 — SSD would take 2.2 days to break even. Picking a TTL is effectively telling the provider where to store your cache.
More from Infra
- Cerebras releases CS-4 engineering documentary detailing wafer-scale AI system design — draecomino · 2026-11-10
- Semi-Apocalypse podcast: market panic, enterprise AI, and the 2028 GPU glut theory — BenBajarin · 2026-11-10
- Ben Bajarin: supply chain bottleneck analysis is back at center stage — BenBajarin · 2026-11-10
- Epoch AI's new Inference Performance Index shows GB300 beats H100 by up to 72x — scaling01 · 2026-11-10
- Maryland activist gathers 25,000 signatures to block $5B AI data center at abandoned mall site — Polymarket · 2026-11-10
- When to move from a homegrown FastAPI LLM proxy to a proper gateway — Kev926 · 2026-11-10