Ragged KV cuts Looped LM inference FLOPs by 25% with no retraining
DanielKhashabi · x · 2026-10-12
A fix for Looped LMs: Ouro promises dynamic computation via early exits for easy tokens, but actually computes all loops. The authors propose:
- Ragged KV: −25% inference FLOPs & memory on Ouro, same accuracy, no retraining.
- Global exit objective: true difficulty-aware exits.
More from Infra
- Robotics research goes big: one work reportedly used 64 H200 GPUs — chris_j_paxton · 2026-10-13
- US needs allied datacenters it can't build at home, argues Foreign Policy piece — abhiadesai · 2026-10-13
- Huawei says Ascend has passed Nvidia in China share, deploys 256K-GPU super node — pstAsiatech · 2026-10-13
- CoreWeave Launches Model Distillation Preview to Cut Frontier-Model Costs — altryne · 2026-10-12
- AWS guide: deploy llama.cpp decision models on SageMaker, GPU cuts tail latency 32x — AWS ML Blog · 2026-10-12
- NVIDIA Opens Vera Rubin Rack Designs, Publishes Open Standards for Next-Gen AI Factories — elonmusk · 2026-10-12