At Scale, KV Cache Becomes a Storage System: How LLMs Serve GBs of Cached State

blaizedsouza · x · 2026-09-10

A technical deep-dive (a popular LLM interview question) on how production LLMs handle gigabytes of KV cache:

Every attention layer creates KV tensors while processing a prompt; during generation the model reads this state instead of re-processing the sequence for each new token. For a single request it's temporary GPU data.
But when multiple requests and inference workers reuse the same prompt prefix, cached data must survive after the original request finishes — even after the creating engine restarts.
A cache hit then requires locating relevant blocks, bringing them back to GPU memory, evicting old blocks, and letting multiple workers share data: the cache effectively becomes a storage system.
The article explains KV, Prefix, Prompt and Semantic caching from first principles, paired with LMCache (11.7k GitHub stars), an open-source KV cache layer for LLMs.

More from Infra