How ResidualQuant works: share a 4-bit anchor, store each loop's KV residual in 2 bits
sehoonkim418 · x · 2026-10-12
- Technical follow-up on ResidualQuant. Key insight: KV states across loops are highly similar; naively quantizing each to low bits loses too much information, but reconstructing one loop from another leaves residuals with fewer outliers that are much easier to represent in few bits.
- The method (two steps per the diagram): (1) use the final loop's KV as a shared anchor and subtract it from each loop to get its residual; (2) store the anchor in INT4 and each residual in INT2, reconstructing by addition.
- Combined with the earlier post: BF16-level accuracy preserved with 80% less KV storage and 3-4x peak decode throughput.
More from Infra
- Ben Bajarin: AI Has Moved the Memory Floor — Stop Valuing DRAM by Old Cycles — BenBajarin · 2026-10-13
- NVIDIA and Cisco unveil USFP optics: 400Gb/s per lane, 4x bandwidth density — BenBajarin · 2026-10-13
- Robotics research goes big: one work reportedly used 64 H200 GPUs — chris_j_paxton · 2026-10-13
- US needs allied datacenters it can't build at home, argues Foreign Policy piece — abhiadesai · 2026-10-13
- Huawei says Ascend has passed Nvidia in China share, deploys 256K-GPU super node — pstAsiatech · 2026-10-13
- CoreWeave Launches Model Distillation Preview to Cut Frontier-Model Costs — altryne · 2026-10-12