Trimming MTP draft vocab to 47k boosts DGX Spark code decoding by 21.5% on same hardware
MaziyarPanahi · x · 2026-09-10
A hands-on optimization for Qwen3.8-Flash-Next (NVFP4) on a single NVIDIA DGX Spark: trimming the MTP speculative-decoding draft vocabulary from 248k rows to a code-tuned 47k shrank the draft head from 1.18 to 0.22 GiB and skipped 2.9 GiB per speculative step. Result: single-stream code decoding jumped from 50.6 to 61.5 tok/s (+21.5%), 18% average single-stream gain, 252 tok/s aggregate at 8 streams — with gains decaying as concurrency rises.
More from Infra
- A100 rental prices still rising in 2026 per a16z data, Jevons paradox hits compute — VraserX · 2026-10-27
- Anthropic S-1 reveals $800B+ in off-books future compute commitments — GaryMarcus · 2026-10-27
- ~42% of Anthropic's ~11GW contracted compute is NVIDIA GPUs — ns123abc · 2026-10-27
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- RTX 5080 price jumps from $1658 to $2318 in ten days on Amazon — Frizzy-MacDrizzle · 2026-10-27
- ComfyUI on 12GB VRAM: measured peaks show offloading turns 24s into 11 minutes — Impossible-Mail-4619 · 2026-10-27