LowRAM: Huawei's KV cache compression gets 4x near-lossless value amortisation
Teresa Delgado · hf · 2026-11-09
A paper from Huawei Noah's Ark Lab (London) and UCL introduces LowRAM, a complementary axis for KV cache compression:
- Key insight: within each attention head, values are largely predictable from their corresponding keys
- Value cache amortisation: a lightweight per-head predictor at test time stores residuals only for tokens it reconstructs worst, yielding 4x near-lossless value compression
- The predictor can be meta-learned to cut prefill overhead, or fitted from scratch in settings allowing one-off cost (e.g. multi-query inference)
- Paired with cross-layer low-rank key compression, composable with quantization and eviction
Motivation: KV cache grows linearly with context length, capping batch size and decoding throughput.
More from Infra
- Impatient waiting for the electrician, ML engineer grabs a few B200s from Lambda — TheZachMueller · 2026-11-09
- Google launches two new TPU variants to challenge Nvidia, reportedly ~30% cheaper — CurieuxExplorer · 2026-11-09
- NVIDIA, Rebellions and SK Telecom partner on Korea's sovereign AI infrastructure — pstAsiatech · 2026-11-09
- DHH: Omarchy 4.6 auto-picks best local models, AMD 495 runs Flash Next impressively — AnushElangovan · 2026-11-09
- Google expands 8th-gen TPU lineup with TPU 8i Max and TPU 8tx, four systems in one generation — rakyll · 2026-11-09
- Autopilot for CI launches: adaptive multi-provider runner routing cuts bills 30% — emeka_boris · 2026-11-09