Why KV cache stores K and V but never Q, clearly explained
AccBalanced · x · 2026-11-18
Akshaypachaar's article and infographic answer a popular LLM interview question: why does the KV cache keep K and V but never Q? After prefill, only the last position's hidden state produces the logits for the next token; computing it needs the newest Q plus every earlier K and V. Old K and V vectors never change across generation steps, so they're cached and reused, while each Q is used exactly once and discarded — hence the cache stores only K and V, appending one new row per step. The infographic walks through prefill, score+mix and cache growth with a worked example.
More from Infra
- Stratechery: Nvidia's earnings are a statement; Microsoft and Anthropic ship dueling AI OS updates — Stratechery · 2026-11-18
- Codeberg goes dark: domain vanished, pages unconfigured — jedisct1 · 2026-11-18
- All hyperscalers but Microsoft post negative FCF as AI capex peaks — marlene_zw · 2026-11-18
- Codeberg's domain mysteriously vanishes from its registrar account, site down — kingslyj · 2026-11-18
- Tokkey debuts thumb-sized AI hardware to run local models on your Mac — TejasKumar_ · 2026-11-18
- Hugging Face ships caching and serving upgrades, TTFT 1.5x faster — osanseviero · 2026-11-18