NVIDIA's Five Guidelines for Speculative Decoding: EAGLE-3, MTP and More Compared
NVIDIAAI · x · 2026-09-05
An NVIDIA technical blog explains how speculative decoding — a small draft model proposes multiple tokens that the large target model verifies in parallel — accelerates LLM inference without sacrificing accuracy, with five practical guidelines:
- Push GEMMs into the compute-bound region without increasing KV cache pressure
- When attention dominates, set draft length to 128/G - 1
- Align larger draft lengths with attention kernel tile boundaries of 128
- At very low latency, increase draft length only while acceptance gains justify the draft cost
- Choose the draft mechanism by balancing acceptance length, draft overhead, and training/deployment cost
It compares external draft models, EAGLE-3, MTP, DFlash, DSpark and suffix/n-gram methods on training cost, serve-time memory and speculation cost. NVIDIA also open-sourced SPEED-Bench for measuring acceptance length on realistic coding/summarization workloads, and ships ready-to-run EAGLE-3, DFlash and DSpark training examples — including fine-tuning and quantization on Nemotron 3.5 Lightning — in the NVIDIA/Model-Optimizer repo.
Related event: NVIDIA Shares Five Rules for Speculative Decoding to Speed Up LLM Inference(4 posts)→
More from Infra
- DeepSeek's token density yields ~$38/GPU-hr revenue vs $3.50 for raw IaaS, engineer estimates — bookwormengr · 2026-10-21
- Sizing a 1 GW datacenter: ~522,000 Blackwell GPUs fit inside — bookwormengr · 2026-10-21
- At 85% billable uptime, a 1 GW fleet supplies 3.89B GPU-hours/year — bookwormengr · 2026-10-21
- DeepSeek's production density reportedly hits ~$38 per GPU-hour at saturation — bookwormengr · 2026-10-21
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21
- Indie dev replaces $1k OpenAI bill with a YAML pipeline on spot GPUs and vLLM — cmm324 · 2026-10-21