ResidualQuant: 2-bit residuals cut Looped Transformer KV cache 80%, 3.3x decode

sehoonkim418 · x · 2026-10-12

KAIST's SEAL lab introduces ResidualQuant, tackling the KV cache bottleneck of Looped Transformers with 2-bit residual quantization.

Problem: Looped Transformers share weights across loops to add depth, but each loop produces its own KV cache—memory still grows with recurrent depth, context length, and batch size.
Method: Exploiting strong similarity across loops, it stores the final loop as a shared INT4 anchor, predicts earlier loops via least-squares scaling, and encodes only the residual in INT2; shared rotations, loop-wise precision allocation, and mixed precision remove the need for a separate BF16 region.
Results: On Ouro-1.4B at 8k context, logical KV storage drops 80.7%, peak decode throughput hits 3.27×, while recovering BF16-level accuracy (76.0% on MATH500).

Paper is out; code coming soon.

More from Infra