arXiv: Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models
rohanpaul_ai · x · 2026-11-03
arXiv link post for the Cohere paper (arXiv:2610.38981, submitted Oct 28, 2026) by Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, and Acyr Locatelli.
Abstract highlights: prior work predicted a concave speedup-batch size relationship for MoE speculative decoding, with gains peaking at intermediate batch sizes due to expert saturation. Evaluating 10 sparse MoE models (30B–1T params, 8–512 experts), the authors find fine-grained models show higher speedup at low batch size. Verification cost depends on temporal overlap across draft tokens and GPU execution dynamics, not just activated-expert bytes: routing overlap reduces unique experts activated, and an IPF framework predicts unique expert counts with under 4% error.
More from Infra
- KV cache translator proposal: small model reads 100K tokens, large model corrects — dreamai87 · 2026-11-03
- HF tokenizers cuts special-token memory 1,000×, boosts decode speed 26% — art_zucker · 2026-11-03
- rauchg: stateless sandboxes are the wrong benchmark as Vercel Sandbox gets 2.4x faster — cramforce · 2026-11-03
- US data center construction spending hits $96.5B/month, 2x general offices — BigBourgeoisie · 2026-11-03
- Microsoft Cloud hits $63B quarter at 30% growth as AI buildout keeps scaling — thedealdirector · 2026-11-03
- Open-source sparse attention: Qwen 27B with no accuracy loss, 10M-token context — Immediate_Sky_6566 · 2026-11-03