BellowsKV shares one latent across layers, keeping 97-98% accuracy at 13-14x KV cache compression
burkov · x · 2026-10-30
Burkov breaks down the Rice University paper BellowsKV (One Latent for Many Layers, Expanded on Demand):
- Problem: For long contexts and multi-step agent tasks, per-layer KV caches can consume as much memory as model weights, limiting serving concurrency; existing compression (quantization, token eviction, dimensionality reduction) degrades sharply at high ratios.
- Key findings: Adjacent layers' KV caches differ substantially, but their input hidden states are highly similar; under the same memory budget, shrinking the shared latent's dimension loses far more accuracy than quantizing it.
- Method: Training-free — maps hidden states of n adjacent layers to one shared latent per token; each layer reads its own keys/values via a layer-specific projection. The latent stays full-dimension and is compressed via low-precision storage (Quant) or one eviction decision per block (Evict).
- Results: Across Llama-3.1-8B, Mistral-7B and Qwen2.5-7B, BellowsKV-Quant retains 99% of full-cache accuracy at 3.6x compression and 97-98% at 13-14x, vs at most 84% at 16x for the strongest prior cross-layer method.
- Code: github.com/MorrisZJ/BellowsKV
More from Infra
- Cap moves transcription to its own GPU farm, up to 11x faster — JosephJacks_ · 2026-10-30
- Prime Intellect makes DeepSeek's MegaMoE trainable, up to 55% faster MoE training — samsja19 · 2026-10-30
- Balanced-ternary K5 inference beats TorchAO FP8 by 3.2× on NVIDIA L4 — Code_MasterCody · 2026-10-30
- Is a $420 Xeon X99 box with 256GB RAM worth it for LLM inference? — Ammoryyy · 2026-10-30
- Meta loses a node every 5 hours; topology-aware scheduling recovers 27% more GPUs — skypilot_org · 2026-10-30
- llmman: run Claude Code, Codex and other agents on any local model with one command — weskambale · 2026-10-30