Speculative decoding has evolved twice: from small-batch speedups to lifting MoE compute intensity
YouJiacheng · x · 2026-09-10
YouJiacheng outlines how the role of speculative decoding has shifted through two evolutions:
- Origin: originally a speedup for dense models at small batch sizes, trading throughput compared to large-batch inference.
- First evolution: it raises the arithmetic intensity of long-context full attention, improving both speed and throughput.
- Second evolution: with sparse attention bounding batch size by capacity and MoE layers running at very low arithmetic intensity, speculative decoding can lift MoE's compute intensity, again boosting both speed and throughput.
The takeaway: speculative decoding has gone from a low-concurrency latency trick to a tool for improving hardware utilization in sparse-attention and MoE architectures.
More from Infra
- Pocket AI Lab: open-source iOS app runs LLMs fully on-device with three backends — Ammoryyy · 2026-09-10
- Triton creator Phil Tillet on Gluon: handing GPU decisions back to AI models — TheTuringPost · 2026-09-10
- DeepSeek open-sources deepseek-recipe to ease deploying V4.1 Flash and future models — teortaxesTex · 2026-09-10
- Qwen 27B with vision on a 16GB GPU: 85K context at 45 tok/s, full config — FerLuisxd · 2026-09-10
- CEA Encoder-Decoder Split Could Reshape GPU Pooling and Heterogeneous Inference — metmelo · 2026-09-10
- Millie's ternary 35B MoE hits 56% on SWE-bench Verified, runs local coding agent on iPhone 17 Pro — MannyKayy · 2026-09-10