Cohere paper: adjacent tokens share 20-45% of experts, making speculative decoding cheap even at batch size 1
rohanpaul_ai · x · 2026-11-03
A new Cohere paper finds speculative decoding works far better on fine-grained MoE models at low batch sizes than prior work predicted.
Key findings
- Tested 10 sparse MoE models (30B–1T params, 8–512 experts) with H100 kernel profiling.
- Adjacent tokens share 20–45% of their experts, versus 4–12% expected under independent routing, so verification activates far fewer unique experts.
- At low batch size, MoE layers take a much smaller share of runtime than of weights: MoE kernels use GPU bandwidth more efficiently than attention, and L2 absorbs repeated reads of co-routed experts.
- Qwen3-235B in FP8 ran its verification pass at 89% of a normal decoding step's speed, versus 52% predicted.
- A maximum-entropy IPF model predicts unique expert counts from routing-overlap statistics with under 4% error.
Practical advice: for low-batch MoE serving like agent workloads, benchmark speculative decoding on your own hardware instead of ruling it out.
More from Infra
- A year of renting an H100 costs $37.6K as warnings grow about tightening compute supply — teortaxesTex · 2026-11-03
- Mistral Large 4 (1T params) runs locally on two M3 Ultras at 20 tok/s via MLX-VLM — andrejusb · 2026-11-03
- AEHR jumps 16% and LSCC 9% after hours as both chip stocks report earnings — BenBajarin · 2026-11-03
- RidgeFill: training-free prefill skipping cuts TTFT 1.6x, 64K request 82.6s to 47.5s — Public_Umpire_1099 · 2026-11-03
- Stanford physicist argues space compute cooling is feasible with a black-sphere thought experiment — shaunmmaguire · 2026-11-03
- Interactive map of AI debt: 25 parties, 58 obligations, and no real counterparty — W1RELESS · 2026-11-03