KAIST's ResidualQuant cuts KV cache storage 80.7% with 2-bit residuals for Looped Transformers
sehoonkim418 · x · 2026-10-12
Researchers at KAIST's SEAL lab (project lead Heejun Kim, with Sehoon Kim, Insu Han, et al.) published ResidualQuant, a KV cache quantization method for Looped Transformers.
Problem: Looped Transformers share weights across depth but each loop produces its own KV cache, so memory still scales with recurrence depth, context length, and batch size.
Method:
- Exploit strong inter-loop KV similarity: store the final loop as a shared INT4 anchor
- Predict earlier loops via least-squares scaling and encode only the residuals in INT2
- Shared rotations and loop-wise mixed-precision allocation preserve loop-specific information without a BF16 region
Results: On Ouro-1.4B (MATH500), ResidualQuant recovers BF16-level accuracy with 80.7% lower logical KV storage and hits 3.27× peak decode throughput at 8k context on an RTX 5090. Code coming soon.
More from Infra
- Ben Bajarin: AI Has Moved the Memory Floor — Stop Valuing DRAM by Old Cycles — BenBajarin · 2026-10-13
- NVIDIA and Cisco unveil USFP optics: 400Gb/s per lane, 4x bandwidth density — BenBajarin · 2026-10-13
- Robotics research goes big: one work reportedly used 64 H200 GPUs — chris_j_paxton · 2026-10-13
- US needs allied datacenters it can't build at home, argues Foreign Policy piece — abhiadesai · 2026-10-13
- Huawei says Ascend has passed Nvidia in China share, deploys 256K-GPU super node — pstAsiatech · 2026-10-13
- CoreWeave Launches Model Distillation Preview to Cut Frontier-Model Costs — altryne · 2026-10-12