ResidualQuant KV Cache Compression Boosts Decode Throughput Up to 4.15x on RTX 5090
sehoonkim418 · x · 2026-10-12
ResidualQuant, a KV cache compression method, was benchmarked on Ouro-1.4B with an RTX 5090: compressing the KV cache cuts per-request memory traffic and frees GPU memory to serve more concurrent requests. Peak decode throughput rises 3.27x at 8K context and 4.15x at 16K context versus the BF16 baseline. The work is by Heejun Kim (project lead), Junyoung Lee, Bill Cho, Dongsu Han, and Insu Han — the tweet author's lab's first paper with his own students.
More from Infra
- The top two US counties for data centers are also the two richest — is there a flywheel? — gregmushen · 2026-10-12
- Cloudflare ships Logpush Transformers: SQL filtering at 1.6M rows/sec with DataFusion — Cloudflare Blog · 2026-10-12
- Local DeepSeek on GPU+CPU+NVMe: 21x prefill speedup and 95% TTFT reduction after weeks of tuning — HankYeomans · 2026-10-12
- Palo Alto Networks Acquires Portkey, the AI Gateway Passing Every Enterprise Prompt and Dollar — YvesMulkers · 2026-10-12
- Anthropic's Compute Spree: Ex-Bitcoin Miners IREN, TeraWulf Join $530B Infrastructure Push — tengyanAI · 2026-10-12
- Random Gemini 400 errors on Cloudflare Workers traced to HKG datacenter routing — sujingshen · 2026-10-12