ResidualQuant: 2-bit KV cache quantization lifts MATH500 accuracy from 27.8% to 76%
sehoonkim418 · x · 2026-10-12
Researchers propose ResidualQuant, a 2-bit KV cache quantization method built on the idea of "share the pattern, store the difference": the final loop of a recurrent model acts as a shared anchor, and each loop stores only its 2-bit residual against it. Reconstruction simply adds the anchor back.
The method combines residual quantization, scaling, rotations, and loop-wise mixed precision to preserve accuracy.
Key results:
- Accuracy: On Ouro-1.4B MATH500, direct INT2 scores 27.8% vs 76.0% with ResidualQuant, beating the prior SOTA OptR-H, while using 80.7% less KV memory than BF16.
- Throughput: On RTX 5090, peak decode throughput improves 3.27× at 8K context and 4.15× at 16K vs BF16, with up to 4× batch capacity.
More from Infra
- Ben Bajarin: AI Has Moved the Memory Floor — Stop Valuing DRAM by Old Cycles — BenBajarin · 2026-10-13
- NVIDIA and Cisco unveil USFP optics: 400Gb/s per lane, 4x bandwidth density — BenBajarin · 2026-10-13
- Robotics research goes big: one work reportedly used 64 H200 GPUs — chris_j_paxton · 2026-10-13
- US needs allied datacenters it can't build at home, argues Foreign Policy piece — abhiadesai · 2026-10-13
- Huawei says Ascend has passed Nvidia in China share, deploys 256K-GPU super node — pstAsiatech · 2026-10-13
- CoreWeave Launches Model Distillation Preview to Cut Frontier-Model Costs — altryne · 2026-10-12