Why KV Cache Eats GPU Memory: The Hidden Cost of Context Length and Concurrency
techNmak · x · 2026-09-25
The author explains why KV cache becomes a dominant GPU memory consumer in serving systems: model weights stay fixed during inference, but active-request state keeps growing — every cached token leaves key/value tensors across attention layers, alive as long as that context is in use.
Key points:
- Context length isn't just a spec number; it has a direct memory cost. Concurrency compounds it: each concurrent request carries its own KV state, so longer sequences and more simultaneous requests compete for the same memory.
- Architecture choices show up immediately: GQA/MQA reduce KV heads, KV quantization shrinks stored bytes, sliding-window attention limits retained history.
- Caching doesn't eliminate attention over existing context — it saves recomputing old K/V projections, but each new query still attends over cached keys and values.
Conclusion: KV cache sits at the intersection of context length, memory capacity, decode cost, and serving concurrency — a small piece of the attention equation with large systems consequences.
More from Infra
- Theoretical Neuroscience podcast: how neuromorphic computing could speed up AI and cut energy use — neurovium · 2026-09-25
- Goldman: Big Tech AI capex to jump 50%+ to $1.2T in 2027, $1.4T in 2028 — rohanpaul_ai · 2026-09-25
- Robot simulation is the cleanest anti-gaming incentive, argues Bittensor SN49 — bittingthembits · 2026-09-25
- SemiAnalysis: NVIDIA B200 serving DeepSeek can yield up to $15B annual profit per gigawatt — zephyr_z9 · 2026-09-25
- Qdrant open-sources Supernova embedding tool and 10B FineWeb dataset — qdrant_engine · 2026-09-25
- Founder argues datacenter buildouts are a dead end — AI models will shrink like mainframes did — draginol · 2026-09-25