NVIDIA's official speculative decoding guide is dominated by Chinese research teams
新智元 · wechat · 2026-09-03
NVIDIA's developer blog published a long-form guide on co-designing AI models with speculative decoding, laying out a comparison table of 6 mainstream inference acceleration methods with training costs and use cases — a rare case of a chip vendor writing hardware-driven model design rules, with nearly all featured methods led by Chinese teams.
The article breaks down the speedup formula (L×Ttarget/(Tdraft+Tverify)) and the evolution: external draft models (1T-10T tokens to train, now paired with NVIDIA's $20B Groq LPU acquisition); EAGLE-3 (Peking U/Waterloo, three generations, now overtaken); MTP (Meta-created, standardized by DeepSeek-V3, NVIDIA's pick for GPUs); DFlash (diffusion-inspired parallel prediction of 7 tokens, 6x lossless speedup); DSpark (DeepSeek+PKU 24-person team, acceptance length 30% above EAGLE-3 and 18% above DFlash, 60-85% faster than MTP in DeepSeek-V4 production).
Key takeaway: hardware constants lock model architecture — from GPU Tile Size 128 comes D=128/G−1, so attention group count G dictates draft length (G=8→15 drafts, G=32→3). The inference efficiency race has shifted from parameter scaling to mastering silicon limits.
More from Infra
- Tencent Hunyuan 770B compressed from ~1.5TB to ~214GiB with mixed quantization — Aiden_Tech_Ai · 2026-09-03
- Cloudflare's Cache Transcoding shrinks cached text assets to ~1/3 with Zstandard — arpit_bhayani · 2026-09-03
- Microcenter shelf suddenly stocked with dozens of RTX 5090s — is the GPU shortage easing? — OvertaxedOne · 2026-09-03
- JapanFold offers 8 free open-source biology AI models on Japan-hosted inference infra — DavidBennett__ · 2026-09-03
- Squeezing Qwen 35B on an RX 6700 XT: a llama.cpp 100k-context tuning log — Loose_Doubt367 · 2026-09-03
- Open-sourced an experimental standalone DLSS 5 video player for neural rendering — 2600th · 2026-09-03