ResidualQuant compresses Looped Transformer KV cache: 80% less memory, 3-4x decode throughput
sehoonkim418 · x · 2026-10-12
- The problem: Looped Transformers reuse weights across loops for extra compute, but each loop produces its own K/V states, so KV cache grows linearly with loop count — the chart shows Ouro-1.4B (8,192 tokens, 24 layers) hitting 103.1 GB of KV memory at 4 loops with batch 16, vs just 2.87 GB of weights. Longer contexts and larger batches amplify the cost, limiting concurrency.
- ResidualQuant: Since KV states are highly similar across loops, it uses the final loop as a shared 4-bit anchor and stores each loop's residual difference in just 2 bits; reconstruction is simply anchor + residual.
- Results: Residuals have fewer outliers and quantize cleanly, preserving BF16-level accuracy with 80% less KV storage and 3-4x peak decode throughput.
More from Infra
- OpenAI chip head Richard Ho on Jalapeno: why Broadcom beat Nvidia and AMD as partner — richardjnieva · 2026-10-12
- SpaceX building $250M GigaBay at Starbase to build up to 1,000 Starships a year — DimaZeniuk · 2026-10-12
- Bittensor's TPN Labs ships first model TPN005, cutting base size ~40% with 32k context intact — const_reborn · 2026-10-12
- Lightning AI launches BYOC: run Studios on your own AWS, GCP, Azure or Slurm infra — LightningAI · 2026-10-12
- Microsoft to scale in-house Maia AI chips to multi-gigawatt inference by 2027 — The Decoder · 2026-10-12
- Microsoft plans huge deployment of its own AI chips to cut costs — pstAsiatech · 2026-10-12