Speculative decoding has evolved twice: from small-batch speedups to rescuing MoE compute intensity
chain_rules · x · 2026-09-11
A technical discussion tracing how the role of speculative decoding has shifted.
- Origin: it started as a way to speed up dense models at small batch sizes, trading throughput for latency.
- First evolution: with long-context full attention, speculative decoding raises arithmetic intensity, improving both speed and throughput.
- Second evolution: sparse attention caps batch size by capacity and MoE runs at very low compute intensity, so speculative decoding now boosts MoE utilization for speed plus throughput.
A reply asks whether the two evolutions map to the Tri Dao et al. paper and the DSpark paper.
Related event: Speculative Decoding's Two Evolutions: From Dense Speedup to MoE Savior(2 posts)→
More from Infra
- RTX 5090 hits $7K as PC gaming community goes into 'denial' over AI boom — PAstynome · 2026-10-27
- Wan2GP lands on Pinokio: 1-click AI video on 6GB GPUs — cocktailpeanut · 2026-10-27
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27
- Thieves spoofing GPS steal $100M+ in data center cargo since June — Ars Technica AI · 2026-10-27
- Lam Research: weekly token usage up ~250x in 18 months — Beth_Kindig · 2026-10-27
- AWS shows how to build enterprise agentic retrieval on Bedrock with CDK — AWS ML Blog · 2026-10-27