A First-Principles Handbook on KV Cache: From MHA/GQA/MLA to PagedAttention

techNmak · x · 2026-09-06

The author compiled a technical handbook on KV cache in LLM inference, built from first principles:

• Core mechanism: during autoregressive generation, each decoding step would otherwise recompute key/value states for already-processed tokens; KV caching stores and reuses those tensors.
• Stack-wide impact: the cache grows with sequence length, consumes significant GPU memory, and creates bandwidth pressure during decoding — explaining the shift from MHA to MQA/GQA, MLA's alternative compression approach, and why systems like PagedAttention exist.
• Coverage: what exactly gets cached, tensor shapes, the memory formula, concrete MHA/GQA/MQA calculations, prefill vs decode, MLA, PagedAttention, prefix reuse, offloading, quantization, eviction, and common misconceptions.

Grounded in original papers and current framework docs.

Original post →

More from Infra

• Book-length deep dive: virtual memory from first principles — page tables, TLBs, NUMA and performance — abhi9u · 2026-09-06
• Qwen3.8 Flash hits 45 tok/s on M4 Max, matching Qwen3.8 27B speed on Apple Silicon — DerTomsn · 2026-09-06
• Altman: 38,000 ChatGPT queries use as much water as one California almond — victor_explore · 2026-09-06
• The best local model you can run on 2 GB10s, per this desk setup — jasonkneen · 2026-09-06
• One ComfyUI node fixed MiniMax H3 OOM on RTX 5090: full workflow for 15s 2K video — denizbuyukayak · 2026-09-06
• KV cache often spills out of HBM in the agentic era, tanking effective bandwidth — AccBalanced · 2026-09-06

Infra channel →