NVIDIA's NarrowSpec: Halving Draft Hidden Width Still Speeds Up Speculative Decoding 1.44x
nvidia · hf · 2026-11-20
NVIDIA shows in NarrowSpec that draft models borrowing the target's hidden states (DFlash, EAGLE-3) don't need to inherit its hidden width—the draft's residual stream can run at half or quarter width while largely preserving acceptance.
- For DFlash, which borrows the target's vocabulary projection, the trained output adapter folds into it, yielding one narrower projection with logits unchanged up to floating-point rounding.
- Tested on Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4) targets in SGLang on a B200 and DGX Spark (GB10): at batch 1, temperature 0, end-to-end decoding speeds up 1.06–1.12x on B200 and 1.13–1.44x on GB10.
- The speedup holds at large batch sizes: up to 1.21x from batch 32 upward, and no narrow draft is ever slower than full width.
More from Infra
- BitDecoding-FP4: native FP4 KV cache decoding via algorithm-system co-design — kalyan_kpl · 2026-11-20
- Prefill beats decode: SGLang wins agent runs on DGX Spark despite slower tok/s — AIFlow_ML · 2026-11-20
- Nvidia supplier commitments jump from $87B to $203B as forecasts miss low — tengyanAI · 2026-11-20
- Qwen3.8-Flash-Next on a single Intel Arc B70: 2.5K prefill, 47 tok/s — WizardlyBump17 · 2026-11-20
- SQLite is ~35% faster than the filesystem for 10KB blobs — here's why the syscall overhead adds up — arpit_bhayani · 2026-11-20
- Tencent Hy4 lands in vLLM day-0: 774B MoE with 49B active, 1M context, 8-GPU FP8 recipe — vllm_project · 2026-11-20