Prefix caching breaks on reordered documents, while CacheBlend keeps about 95% cached

blaizedsouza · x · 2026-07-21

Why prefix caching breaks and how CacheBlend fixes it

The post argues that most KV cache entries in real LLM systems never get reused, and that plain prompt caching is not enough once prompts become dynamic.

Key points:

Prompt caching can cut cached input token cost by up to 90%, with realistic hit rates of 60–85%.
But it only works when the cached text is an exact byte-for-byte prefix of the next request.
That breaks in common cases like multi-document RAG, changing document order, and other prompt permutations.
The attached diagram contrasts prefix caching with CacheBlend, which recomputes only boundary tokens so the cache can survive reordering.
The claim is that this yields much higher reuse, roughly 95% cached, and can be 2–4× faster.

The linked article is framed as a practitioner’s guide to better KV cache management and lower inference cost.

Original post →

Prefix caching breaks on reordered documents, while CacheBlend keeps about 95% cached

More from Infra