NVIDIA's Five Guidelines for Speculative Decoding: EAGLE-3, MTP and More Compared
NVIDIAAI · x · 2026-09-05
An NVIDIA technical blog explains how speculative decoding — a small draft model proposes multiple tokens that the large target model verifies in parallel — accelerates LLM inference without sacrificing accuracy, with five practical guidelines:
- Push GEMMs into the compute-bound region without increasing KV cache pressure
- When attention dominates, set draft length to 128/G - 1
- Align larger draft lengths with attention kernel tile boundaries of 128
- At very low latency, increase draft length only while acceptance gains justify the draft cost
- Choose the draft mechanism by balancing acceptance length, draft overhead, and training/deployment cost
It compares external draft models, EAGLE-3, MTP, DFlash, DSpark and suffix/n-gram methods on training cost, serve-time memory and speculation cost. NVIDIA also open-sourced SPEED-Bench for measuring acceptance length on realistic coding/summarization workloads, and ships ready-to-run EAGLE-3, DFlash and DSpark training examples — including fine-tuning and quantization on Nemotron 3.5 Lightning — in the NVIDIA/Model-Optimizer repo.
Related event: NVIDIA Shares Five Rules for Speculative Decoding to Speed Up LLM Inference(4 posts)→
More from Infra
- Developer Moves AI Agents to a Dedicated Linux Workstation Using Incus System Containers — kevinkern · 2026-10-21
- Nvidia guides 70% growth citing unmet AI demand, while media panics over a $500M Korean neocloud IPO — firstadopter · 2026-10-21
- Pichai: everyone is massively short on AI compute — 'everyone wishes you had two times what you had planned' — firstadopter · 2026-10-21
- Stanford CS336 Lecture 6: fused Triton kernels cut softmax memory traffic ~4x — stanfordnlp · 2026-10-21
- General Compute partners with Positron AI: Atlas memory-centric hardware ships Q1 2027 — aftahi_ai · 2026-10-21
- AI bond issuance hits $419B in 2026, up 90% from all of 2025 — SumitGup · 2026-10-21