Prefix caching breaks on reordered documents, while CacheBlend keeps about 95% cached
blaizedsouza · x · 2026-07-21
Why prefix caching breaks and how CacheBlend fixes it
The post argues that most KV cache entries in real LLM systems never get reused, and that plain prompt caching is not enough once prompts become dynamic.
Key points:
- Prompt caching can cut cached input token cost by up to 90%, with realistic hit rates of 60–85%.
- But it only works when the cached text is an exact byte-for-byte prefix of the next request.
- That breaks in common cases like multi-document RAG, changing document order, and other prompt permutations.
- The attached diagram contrasts prefix caching with CacheBlend, which recomputes only boundary tokens so the cache can survive reordering.
- The claim is that this yields much higher reuse, roughly 95% cached, and can be 2–4× faster.
The linked article is framed as a practitioner’s guide to better KV cache management and lower inference cost.
More from Infra
- PyTorch details 2.12x faster on-device streaming ASR on Arm SME2 via ExecuTorch — PyTorch · 2026-10-27
- Revenue per GPU can rise as token prices fall, argues Gavin Baker — GavinSBaker · 2026-10-27
- Anthropic S-1: only $19B total compute spend since inception, overbuild warning — bookwormengr · 2026-10-27
- CoreWeave, You.com and NVIDIA lift BrowseComp 22.9% in 3-week RL post-training sprint — RichardSocher · 2026-10-27
- Grid queues make power architecture an AI-factory decision; Exowatt and Redwood offer fixes — ingliguori · 2026-10-27
- Two CVSS 9.9 Wasmtime flaws allow sandbox escape, one ARM64-only since 2020 — basedjensen · 2026-10-27