ResidualQuant KV cache compression boosts decode throughput up to 4.15x on RTX 5090
sehoonkim418 · x · 2026-10-12
ResidualQuant is a KV cache compression method: fewer bytes per request cut memory traffic and free GPU memory to serve more concurrent requests. On Ouro-1.4B with an RTX 5090, it boosts peak decode throughput by 3.27x at 8K context and 4.15x at 16K vs BF16. The author celebrates project lead heejunkim00 and collaborators — also his lab's first paper with its own students.
More from Infra
- Ben Bajarin: AI Has Moved the Memory Floor — Stop Valuing DRAM by Old Cycles — BenBajarin · 2026-10-13
- NVIDIA and Cisco unveil USFP optics: 400Gb/s per lane, 4x bandwidth density — BenBajarin · 2026-10-13
- Robotics research goes big: one work reportedly used 64 H200 GPUs — chris_j_paxton · 2026-10-13
- US needs allied datacenters it can't build at home, argues Foreign Policy piece — abhiadesai · 2026-10-13
- Huawei says Ascend has passed Nvidia in China share, deploys 256K-GPU super node — pstAsiatech · 2026-10-13
- CoreWeave Launches Model Distillation Preview to Cut Frontier-Model Costs — altryne · 2026-10-12