KAIST's ResidualQuant: 2-bit Residual Quantization Cuts Looped Transformer KV Cache by 80%

KAIST's SEAL team (led by Heejun Kim, with Sehoon Kim and others) has published ResidualQuant, a 2-bit residual quantization scheme that tackles KV cache bloat in Looped Transformers, saving roughly 80% of memory.

Confirmed

• Background: Looped Transformers reuse the same set of cross-layer shared weights across multiple loops to gain more compute, but each loop produces its own K/V states, so the KV cache grows linearly with the number of loops; an example configuration, Ouro-1.4B (8192 tokens, 24 layers), already faces considerable cache overhead with just 4 loops.
• The core idea is "share the pattern, store only the differences": KV states across loops are highly similar, and naively compressing all of them to low bits loses information; the method uses the final loop of the recurrent network as a shared anchor (stored at 4-bit), subtracts the anchor from each loop to get residuals stored at just 2-bit, and reconstructs by reading back the anchor plus the residuals.
• Key insight: reconstruct one loop first, then store the difference — the residuals contain fewer outliers, so extremely low bit widths suffice.
• Results: about 80% KV cache savings; accuracy on MATH500 rises from 27.8% to 76%, meaning the compression not only avoids degrading quality but is clearly better than the baseline (presumably a naive low-bit quantization baseline).

Why it matters

• Looped / weight-reuse architectures trade small parameter counts for more compute depth, but linear KV cache growth is the key bottleneck to their practicality — ResidualQuant hits that bottleneck head-on.
• The "shared anchor + low-bit residual" approach exploits similarity across recurrent states and offers a useful reference for other cache compression problems involving redundant states.

2026-10-12 ~ 2026-10-12 · 10 related posts

• ResidualQuant: 2-bit residuals cut Looped Transformer KV cache 80%, 3.3x decode — sehoonkim418 · 2026-10-12
• [source] ResidualQuant compresses Looped Transformer KV cache: 80% less memory, 3-4x decode throughput — sehoonkim418 · 2026-10-12
• Loop-similar KV states leave low-outlier residuals, enabling cheap KV cache quantization — sehoonkim418 · 2026-10-12
• How ResidualQuant works: share a 4-bit anchor, store each loop's KV residual in 2 bits — sehoonkim418 · 2026-10-12
• ResidualQuant: 2-bit KV cache quantization lifts MATH500 accuracy from 27.8% to 76% — sehoonkim418 · 2026-10-12
• [source] ResidualQuant on RTX 5090: 4.15x decode throughput at 16K context vs BF16 — sehoonkim418 · 2026-10-12
• ResidualQuant KV Cache Compression Boosts Decode Throughput Up to 4.15x on RTX 5090 — sehoonkim418 · 2026-10-12

3 near-duplicate retellings: sehoonkim418 · sehoonkim418 · sehoonkim418