LeapQuant: training-free quantization makes linear attention up to 4.25x faster
tianyin_xu · x · 2026-11-12
Qwen, Kimi and GLM are all betting on linear attention, but its recurrent state is read and rewritten every token, keeping it slow. LeapQuant is the first training-free method enabling 8-bit or 4-bit recurrent states.
Key results:
- Matches FP32 accuracy
- Linear attention kernels up to 4.25x faster
- Kernel-level speedups of 2.05–3.70x and 1.57x end-to-end on NVIDIA B200, RTX PRO 6000, RTX 5090
- Memory cut by up to 56%
A practical drop-in for deploying linear-attention architectures at inference time.
More from Infra
- Vmax teases Campaign: self-play RL task generation taught agents to use Modal SDK — soleio · 2026-11-12
- JITEngine synthesizes custom LLM inference engines for Apple Silicon, up to 2.7x faster — CShorten30 · 2026-11-12
- Microsoft's MAI Transcribe 2 sees 12x usage growth, hitting 8.5M weekly transcriptions — mustafasuleyman · 2026-11-12
- gVisor joins CNCF with Modal as co-maintainer after upstreaming GPU and training support — josh_wills · 2026-11-12
- Local vs Cloud AI Coding Agents: Benchmarks, Hardware Costs, Performance — luisdans · 2026-11-12
- Agent Substrate 1.0 GA: microVM-sandboxed agent runtime with zero ambient credentials — rakyll · 2026-11-12