Speculative decoding has evolved twice: from small-batch speedups to lifting MoE compute intensity
YouJiacheng · x · 2026-09-10
YouJiacheng outlines how the role of speculative decoding has shifted through two evolutions:
- Origin: originally a speedup for dense models at small batch sizes, trading throughput compared to large-batch inference.
- First evolution: it raises the arithmetic intensity of long-context full attention, improving both speed and throughput.
- Second evolution: with sparse attention bounding batch size by capacity and MoE layers running at very low arithmetic intensity, speculative decoding can lift MoE's compute intensity, again boosting both speed and throughput.
The takeaway: speculative decoding has gone from a low-concurrency latency trick to a tool for improving hardware utilization in sparse-attention and MoE architectures.
Related event: Speculative Decoding's Two Evolutions: From Dense Speedup to MoE Savior(2 posts)→
More from Infra
- Analyst: Ant has the most fungible AI workload of any model provider today — BenBajarin · 2026-10-26
- Anthropic S-1: training infra edge cut per-task cost 40% and doubled revenue growth — BenBajarin · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- US states halt new data centers as 72% of Americans oppose local builds — YvesMulkers · 2026-10-26
- $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
- HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26