KAIST's ResidualQuant cuts KV cache storage 80.7% with 2-bit residuals for looped transformers
sehoonkim418 · x · 2026-10-12
Researchers at KAIST's SEAL lab (with Yonsei and SNU) published ResidualQuant, a KV cache quantization method for looped transformers.
Problem: Looped transformers share weights across depth, but each loop produces a distinct KV cache, so memory still scales with recurrence depth, context length, and batch size—capping the throughput gains weight sharing should enable.
Method:
- Exploits strong inter-loop KV similarity: stores the final loop as a shared INT4 anchor
- Predicts earlier loops via least-squares scaling and encodes only the remaining residuals in INT2
- Shared rotations and loop-wise mixed-precision allocation preserve loop-specific information without a BF16 region
Results: On Ouro-1.4B / MATH500, it recovers BF16-level accuracy with 80.7% lower logical KV storage and hits 3.27× peak decode throughput at 8k context (RTX 5090). Code coming soon.
More from Infra
- Ben Bajarin: AI Has Moved the Memory Floor — Stop Valuing DRAM by Old Cycles — BenBajarin · 2026-10-13
- NVIDIA and Cisco unveil USFP optics: 400Gb/s per lane, 4x bandwidth density — BenBajarin · 2026-10-13
- Robotics research goes big: one work reportedly used 64 H200 GPUs — chris_j_paxton · 2026-10-13
- US needs allied datacenters it can't build at home, argues Foreign Policy piece — abhiadesai · 2026-10-13
- Huawei says Ascend has passed Nvidia in China share, deploys 256K-GPU super node — pstAsiatech · 2026-10-13
- CoreWeave Launches Model Distillation Preview to Cut Frontier-Model Costs — altryne · 2026-10-12