Speculative decoding has evolved twice: from small-batch speedups to lifting MoE compute intensity
YouJiacheng · x · 2026-09-10
YouJiacheng outlines how the role of speculative decoding has shifted through two evolutions:
- Origin: originally a speedup for dense models at small batch sizes, trading throughput compared to large-batch inference.
- First evolution: it raises the arithmetic intensity of long-context full attention, improving both speed and throughput.
- Second evolution: with sparse attention bounding batch size by capacity and MoE layers running at very low arithmetic intensity, speculative decoding can lift MoE's compute intensity, again boosting both speed and throughput.
The takeaway: speculative decoding has gone from a low-concurrency latency trick to a tool for improving hardware utilization in sparse-attention and MoE architectures.
Related event: Speculative Decoding's Two Evolutions: From Dense Speedup to MoE Savior(2 posts)→
More from Infra
- Michael Dell: Musk is "the most demanding customer there is" — Colossus servers built in ~6 weeks — elonmusk · 2026-10-27
- AI chip startup Tenstorrent raises $1.5B at $6.2B valuation, more than doubling — Beth_Kindig · 2026-10-27
- Ex-MediaTek expert: Google TPU external revenue $8B this year, $104B by 2028 — RihardJarc · 2026-10-27
- Anthropic discloses up to $120.5B SpaceX compute deal, ~$3.1B monthly at full capacity — XFreeze · 2026-10-27
- Satire: swapping Raft for an LLM-style decision model tanks write throughput 99.98% — irombie · 2026-10-27
- Cadence joins the domain-specific AI model camp for chip design — karlfreund · 2026-10-27