NVIDIA Research Blog: Why KV Cache Compression Breaks in Production vLLM Deployments
blaizedsouza · x · 2026-11-03
A long-form NVIDIA Research blog (Weian Mao, Song Han et al.) examines KV cache compression through an infrastructure lens: most algorithms work in the lab but fail in real deployments.
The hook is a counterintuitive interview question: a reasoning model served on vLLM keeps OOM-ing on long traces; you compress the KV cache and evict 90% of cached tokens, yet VRAM usage barely moves. The culprit is paged attention — memory is only reclaimed when a block is completely empty, and surviving tokens leave almost every block partially occupied, so the "freed" memory never comes back.
Two infrastructure collisions plague the field:
- Existing methods need historical attention scores, but FlashAttention kernels never write them to GPU memory;
- Paged attention's block-granular reclamation makes token eviction nearly byte-free in practice.
Concrete example: a 4-bit Qwen3-32B on a 24GB GPU OOMs around 24K tokens of a 32K-token reasoning trace, since KV cache dominates memory for long chains of thought. The post surveys the field's main ideas and shows how a geometric property hidden beneath RoPE has recently been used to resolve both problems.
More from Infra
- Analyst: Palantir earnings could hurt the AI trade more than help it, squeezing rivals alongside Nvidia — pdamodaran · 2026-11-03
- Inference providers can't survive without DeepSeek-grade KV-cache engineering, argues thread — AccBalanced · 2026-11-03
- Inference is a craft, not an appliance — many providers won't survive, says founder — AccBalanced · 2026-11-03
- Agent-designed nested Kubernetes horror: kcp partitions on bare metal under KubeVirt under guest clusters — sloppenheimer · 2026-11-03
- RTX 8000 48GB down to $2,100 while 3090 prices doubled — owner calls it the best homelab all-rounder — triynizzles1 · 2026-11-03
- OpenAI built Pando, a Git backend that cut push latency ~80% and serves ~1PB of Git data daily — gdb · 2026-11-03