MHA vs MQA vs GQA vs MLA: same 8K tokens, KV cache differs 32x
techNmak · x · 2026-11-17
A systematic explainer of how modern LLM attention architectures manage the KV cache: MHA caches per-head KV pairs; MQA shares one KV head; GQA shares within groups; DeepSeek-style MLA compresses KV into a shared latent plus a separate RoPE key, absorbing projections during decoding. Example (BF16, 8,192 cached tokens): per-layer cache is 128 MiB for MHA, 4 MiB for MQA, 32 MiB for GQA, and 9 MiB for MLA. Sharing KV heads doesn't equalize attention distributions, and smaller caches don't automatically improve quality or speed—the key distinction is sharing (MQA/GQA) vs compression (MLA), with direct impact on memory, bandwidth, and long-context/concurrent serving.
More from Infra
- SemiAnalysis shows Trainium3 profiling stack is precise enough to render pixel-art videos — joey00072fp8 · 2026-11-17
- Lawmaker questions Commerce: Megaspeed kept buying Nvidia chips after blacklisting — teortaxesTex · 2026-11-17
- India's sovereign AI lab Sarvam hosts mostly Chinese models on subsidized H100s — ayushthakur0 · 2026-11-17
- Nvidia put Megaspeed on its 'no sell' list in June 2025 after US investigators requested sales records — Polymarket · 2026-11-17
- NVIDIA open-sources cuPotential, over 3x faster MD nonbonded interactions vs OpenMM — AllThingsApx · 2026-11-17
- Syzygy says compute shortage is a software problem, claiming 6% to 90% GPU utilization gains — teortaxesTex · 2026-11-17