Loop-similar KV states leave low-outlier residuals, enabling cheap KV cache quantization
sehoonkim418 · x · 2026-10-12
Looping lets a model do more computation with the same weights, but each loop produces its own K/V states, so the KV cache grows with the number of loops — longer contexts and larger batches amplify the cost, limiting concurrency and increasing memory traffic during decoding.
The obvious fix, KV cache quantization, loses too much information at low bit widths. The opportunity: KV states are highly similar across loops. Reconstructing one loop's states from another leaves a residual with far fewer outliers, making it representable in just a few bits. The shared chart shows |K₁| and |K₂| nearly identical while the residual |K₂ − α₂K₁| collapses toward zero, supporting the approach.
More from Infra
- The top two US counties for data centers are also the two richest — is there a flywheel? — gregmushen · 2026-10-12
- Cloudflare ships Logpush Transformers: SQL filtering at 1.6M rows/sec with DataFusion — Cloudflare Blog · 2026-10-12
- Local DeepSeek on GPU+CPU+NVMe: 21x prefill speedup and 95% TTFT reduction after weeks of tuning — HankYeomans · 2026-10-12
- Palo Alto Networks Acquires Portkey, the AI Gateway Passing Every Enterprise Prompt and Dollar — YvesMulkers · 2026-10-12
- Anthropic's Compute Spree: Ex-Bitcoin Miners IREN, TeraWulf Join $530B Infrastructure Push — tengyanAI · 2026-10-12
- Random Gemini 400 errors on Cloudflare Workers traced to HKG datacenter routing — sujingshen · 2026-10-12