Qwen3.6-27B speculative decoding benchmark finds DFlash fastest, with up to 4.6x speedup
thavoc77 · reddit · 2026-07-21
Benchmarks compare speculative decoding methods on Qwen3.6-27B (dense, NVFP4) on a single RTX PRO 6000 Max-Q, using the same client and three restart samples per point across vLLM and SGLang.
Main results
- DFlash is the clear winner: about 3.3x on SGLang and 2.5x on vLLM, reaching 4.6x on math reasoning.
- MTP / NEXTN delivers roughly 2.2x–2.8x, with speedups improving as draft depth increases.
- EAGLE3 peaks around 1.9x at K=3, then flattens out.
- ngram barely helps, only 1.1x–1.3x.
Engineering notes
- EAGLE3 would not load on vLLM for this model because of an hfhub head-dim validator issue; it only worked on SGLang with a patched build.
- DFlash initially crashed on the first token in SGLang because its sampler does a raw matmul on an NVFP4-quantized lmhead; dequantizing the head once fixes it.
- They also had to cap max-running-requests to avoid a mamba/GDN cache OOM.
More from Infra
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11
- Can a 7900 XTX 24GB run Qwen locally? Reddit seeks ROCm tok/s benchmarks — thenomadexplorerlife · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11