Prefix caching breaks on reordered documents, while CacheBlend keeps about 95% cached
blaizedsouza · x · 2026-07-21
Why prefix caching breaks and how CacheBlend fixes it
The post argues that most KV cache entries in real LLM systems never get reused, and that plain prompt caching is not enough once prompts become dynamic.
Key points:
- Prompt caching can cut cached input token cost by up to 90%, with realistic hit rates of 60–85%.
- But it only works when the cached text is an exact byte-for-byte prefix of the next request.
- That breaks in common cases like multi-document RAG, changing document order, and other prompt permutations.
- The attached diagram contrasts prefix caching with CacheBlend, which recomputes only boundary tokens so the cache can survive reordering.
- The claim is that this yields much higher reuse, roughly 95% cached, and can be 2–4× faster.
The linked article is framed as a practitioner’s guide to better KV cache management and lower inference cost.
More from Infra
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22
- Strangeworks launches Aura to turn enterprise ops into production optimization systems — whurley · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- Hybrid and local inference are emerging as a response to AI energy and token costs — dmitry140 · 2026-07-22
- NVIDIA details Vera CPU with 2x performance claims and a 22,000-core rack — ryanshrout · 2026-07-22
- NVIDIA says Vera Rubin NVL72 delivers 10x more tokens per megawatt than Blackwell — nvidia · 2026-07-22