Qwen3.8 Flash hits 74 tok/s single-stream, 212 tok/s aggregate on one DGX Spark — open vLLM recipe
DimeRhyme · reddit · 2026-09-29
Results
The author spent weeks tuning a vLLM serving recipe for Qwen3.8 Flash on a single DGX Spark (GB10), fully open-sourced with raw benchmark data:
- 74 tok/s peak single-stream (typical setups land at 35-45), 60-70 on normal requests, and 212 tok/s across 8 streams.
- Cold prefill 2-3.4x faster than the forked recipe (16K prompt: 4,016 vs 1,171 tok/s).
- Full 262K context; cached coding prompts start replying in 0.57s; quality matches the original within noise (93.1%/93.3% on a 492-question suite, top-1 agreement shift of 0.06 pts).
What made the difference
- Running the model's own MTP head densely: 3.7 tokens per verify step — the biggest lever, with wrong guesses replaced by the full model so quality is unchanged.
- Pruning the draft head vocab from 248K to 65K ids: the draft pass is memory-bound on GB10; the target model still verifies against the full vocab.
- Quantization: W4A16 AutoRound for MoE experts, FP8 for side layers, INT8 lmhead — deliberately avoiding 3-bit/NVFP4 so speed comes from the serving path.
- GB10-tuned low-latency GEMM for decode-time matmuls plus sort-free top-k in verification.
- A faster per-layer embedding gather path removing serial page faults during prefill.
- Decode step went from 68.3ms to 52.3ms on an agent-shaped coding workload.
- A failed experiment: doubling prefill chunk to 16,384 tokens gave no gain and strained memory.
Footprint
71 GiB model, 16 GB KV pool, 16 GiB free under load; GPU at 35-37W median while generating. Most tricks apply to any MTP/speculative-decoding setup.
More from Infra
- Sarvam AI Launches Token Factory: India-Hosted Inference for Open Models — itsOmSarraf_ · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Jailbroken PS4 runs a 9B open-source model at up to 43 tok/s by talking directly to its 2013-era GPU — AIFlow_ML · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Stratechery's Fictional Obituary: Imagining Jensen Huang's 2026 Exit From Nvidia — Gregory_C_Allen · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16