Why KV cache stores K and V but never Q, clearly explained

AccBalanced · x · 2026-11-18

Akshaypachaar's article and infographic answer a popular LLM interview question: why does the KV cache keep K and V but never Q? After prefill, only the last position's hidden state produces the logits for the next token; computing it needs the newest Q plus every earlier K and V. Old K and V vectors never change across generation steps, so they're cached and reused, while each Q is used exactly once and discarded — hence the cache stores only K and V, appending one new row per step. The infographic walks through prefill, score+mix and cache growth with a worked example.

Original post →

More from Infra

Infra channel →