ResidualQuant on RTX 5090: 4.15x decode throughput at 16K context vs BF16
sehoonkim418 · x · 2026-10-12
The ResidualQuant team shares practical gains from KV cache compression: fewer bytes cut memory traffic per request and free GPU memory to serve more requests at once.
On RTX 5090 with Ouro-1.4B vs BF16:
- 4K context: 2.52× peak decode throughput, 4× batch capacity
- 8K context: 3.27× throughput, 4× batch capacity
- 16K context: 4.15× throughput, 2× batch capacity
The longer the context, the bigger the throughput win from compression.
More from Infra
- CoreWeave Launches Model Distillation Preview to Cut Frontier-Model Costs — altryne · 2026-10-12
- NVIDIA Opens Vera Rubin Rack Designs, Publishes Open Standards for Next-Gen AI Factories — elonmusk · 2026-10-12
- Pokee AI: frontier-level models will run fully on local hardware within 2 years — Kyrannio · 2026-10-12
- OpenAI chip head Richard Ho on Jalapeno: why Broadcom beat Nvidia and AMD as partner — richardjnieva · 2026-10-12
- SpaceX building $250M GigaBay at Starbase to build up to 1,000 Starships a year — DimaZeniuk · 2026-10-12
- Bittensor's TPN Labs ships first model TPN005, cutting base size ~40% with 32k context intact — const_reborn · 2026-10-12