What are KV caches really? A storage expert's explainer from prefill to offload
TheZachMueller · x · 2026-09-22
HPC and storage expert Glenn K. Lockwood published "What are KV caches really?", threading the needle between oversimplified analogies and math-heavy formalism by explaining KV caching through what users actually experience.
- Inference has two steps: prefill (the lag before the first token appears) and decode (token-by-token generation);
- Keys and values are intermediate states produced while processing the prefix; caching them lets decode avoid recomputing the whole context;
- The piece extends to KV cache offload, helping less-technical infra folks tell when users might actually hit problems (memory pressure, long contexts, concurrent requests).
Useful context for long-context costs, prompt-cache pricing, and inference serving architecture.
More from Infra
- Un-fusing a realtime voice stack (STT → LLM → TTS) cut costs 14x — and the real win was text-level guardrails — Cloudsurfer_90 · 2026-09-22
- Jensen Huang: A Gigawatt-Scale AI Factory Costs $50-60B, Architecture Must Stay Fungible — nvidia · 2026-09-22
- FreeToken fork adds DeepSeek-V4.1, vision and speculative decoding, with 2x3090 benchmarks — ApeGrower · 2026-09-22
- Bloomberg calls Nvidia's low P/E a warning sign; investor argues it makes the stock more attractive — firstadopter · 2026-09-22
- Eugene Plexus open-source alpha: a browser control plane for multi-machine local inference — ScreamingAmish · 2026-09-22
- Nscale IPO will test Wall Street's appetite for concentrated AI bets — TechCrunch AI · 2026-09-22