NVIDIA's Five Guidelines for Speculative Decoding: EAGLE-3, MTP and More Compared
NVIDIAAI · x · 2026-09-05
An NVIDIA technical blog explains how speculative decoding — a small draft model proposes multiple tokens that the large target model verifies in parallel — accelerates LLM inference without sacrificing accuracy, with five practical guidelines:
- Push GEMMs into the compute-bound region without increasing KV cache pressure
- When attention dominates, set draft length to 128/G - 1
- Align larger draft lengths with attention kernel tile boundaries of 128
- At very low latency, increase draft length only while acceptance gains justify the draft cost
- Choose the draft mechanism by balancing acceptance length, draft overhead, and training/deployment cost
It compares external draft models, EAGLE-3, MTP, DFlash, DSpark and suffix/n-gram methods on training cost, serve-time memory and speculation cost. NVIDIA also open-sourced SPEED-Bench for measuring acceptance length on realistic coding/summarization workloads, and ships ready-to-run EAGLE-3, DFlash and DSpark training examples — including fine-tuning and quantization on Nemotron 3.5 Lightning — in the NVIDIA/Model-Optimizer repo.
Related event: NVIDIA shares five speculative decoding rules for faster LLM inference(3 posts)→
More from Infra
- Back-of-Envelope: Global Compute Equals ~20M H100s; Inference Dominates, Pretraining Far Less Efficient Than Biology — JosephJacks_ · 2026-09-05
- Tenstorrent and aiand Launch JapanFold, Free Open-Source Drug Discovery Models on Galaxy Clusters — DavidBennett__ · 2026-09-05
- Aer Lingus says 50% of long-haul fleet now has Starlink, full coverage by year-end — elonmusk · 2026-09-05
- Running Qwen3.8-27b-mlx Locally on an M5 MacBook to Update a Website and Generate Assets — walkingriver · 2026-09-05
- Agent Substrate brings instant suspend/resume and 10x density to K8s AI agents — davemccollough · 2026-09-05
- Horizontal AI apps plus vertical hardware integration may breed dominant vendors — matt_slotnick · 2026-09-05