At Scale, KV Cache Becomes a Storage System: How LLMs Serve GBs of Cached State

blaizedsouza · x · 2026-09-10

A technical deep-dive (a popular LLM interview question) on how production LLMs handle gigabytes of KV cache:

• Every attention layer creates KV tensors while processing a prompt; during generation the model reads this state instead of re-processing the sequence for each new token. For a single request it's temporary GPU data.
• But when multiple requests and inference workers reuse the same prompt prefix, cached data must survive after the original request finishes — even after the creating engine restarts.
• A cache hit then requires locating relevant blocks, bringing them back to GPU memory, evicting old blocks, and letting multiple workers share data: the cache effectively becomes a storage system.
• The article explains KV, Prefix, Prompt and Semantic caching from first principles, paired with LMCache (11.7k GitHub stars), an open-source KV cache layer for LLMs.

Original post →

More from Infra

• turbovec: Rust vector index fits 10M document vectors in 4GB RAM and outpaces FAISS — tom_doerr · 2026-09-10
• LM Studio 0.4.24 adds advanced llama.cpp argument overrides for GGUF model loading — solyarisoftware · 2026-09-10
• tszzl wraps up: efficiency gains only amplify hunger for hardware — tszzl · 2026-09-10
• Trimming MTP draft vocab to 47k boosts DGX Spark code decoding by 21.5% on same hardware — MaziyarPanahi · 2026-09-10
• Is 5 tokens/s usable for local LLMs? Redditor runs 27B model off an iGPU — Zombiecidialfreak · 2026-09-10
• Deep-Dive Speculative Decoding Blog Incoming: Drafter Training to vLLM Serving — auto_grad_ · 2026-09-10

Infra channel →