One architectural change cuts KV cache 8x: how GQA works, explained with Llama 3 70B
blaizedsouza · x · 2026-09-14
Using Llama 3 70B as an example, the thread explains how Grouped-Query Attention (GQA) slashes KV cache size. Standard MHA stores one KV head per query head, so a 64-query-head model stores 64 KV sets per token per layer. Multi-Query Attention shares a single KV head — smallest cache, but quality suffers. GQA splits the difference: in Llama 3 70B, every 8 query heads share one KV head, storing 8 sets instead of 64 — an 8x smaller cache and 8x less KV data read during decoding, all else equal.
The quoted article, 'KV Cache Engineering for LLM Serving,' covers why the cache grows, 12 ways models and serving engines reduce it, what each technique actually saves, and the trade-offs.
More from Infra
- llama.cpp Adds Maple 20B-A1B Ternary MoE Architecture for CPU and Low-VRAM Devices — jacek2023 · 2026-09-14
- Claude usage boosts quietly removed, fueling talk that 'The Great Compute Crunch has begun' — jacob_posel · 2026-09-14
- Unions urged to halt AI datacenter buildout until jobs and grid use are protected — nordicinst · 2026-09-14
- SK hynix completes HBM4 internal qualification, ushering in custom base die competition — blaizedsouza · 2026-09-14
- A complete breakdown of HBM system architecture, from DDR roots to GDDR7, PIM and HBF alternatives — blaizedsouza · 2026-09-14
- Nvidia paper shows transformer LLMs can be sparser, faster, and lighter without losing accuracy — YesThisIsLion · 2026-09-14