Qwen3.6-27B speculative decoding benchmark finds DFlash fastest, with up to 4.6x speedup
thavoc77 · reddit · 2026-07-21
Benchmarks compare speculative decoding methods on Qwen3.6-27B (dense, NVFP4) on a single RTX PRO 6000 Max-Q, using the same client and three restart samples per point across vLLM and SGLang.
Main results
- DFlash is the clear winner: about 3.3x on SGLang and 2.5x on vLLM, reaching 4.6x on math reasoning.
- MTP / NEXTN delivers roughly 2.2x–2.8x, with speedups improving as draft depth increases.
- EAGLE3 peaks around 1.9x at K=3, then flattens out.
- ngram barely helps, only 1.1x–1.3x.
Engineering notes
- EAGLE3 would not load on vLLM for this model because of an hfhub head-dim validator issue; it only worked on SGLang with a patched build.
- DFlash initially crashed on the first token in SGLang because its sampler does a raw matmul on an NVFP4-quantized lmhead; dequantizing the head once fixes it.
- They also had to cap max-running-requests to avoid a mamba/GDN cache OOM.
More from Infra
- NVIDIA brings its Cosmos 3 Edge world model to Jetson for on-device robot control — liu_mingyu · 2026-07-21
- A silicon photonic reservoir chip compensates fiber distortion in real time at 28 Gbps — bravo_abad · 2026-07-21
- Chamath says open-sourcing Grok would push AI margins from models to infra and apps — Dan_Jeffries1 · 2026-07-21
- EU AI competitiveness is under pressure as firms double down on chips, ethics, and talent — nordicinst · 2026-07-21
- AI bottlenecks are shifting to memory, optics, yield control and power — thedealdirector · 2026-07-21
- llama.garden is using torrents and web seeds to decentralize LLM distribution — de4dee · 2026-07-21