Qwen3.6-27B speculative decoding benchmark finds DFlash fastest, with up to 4.6x speedup
thavoc77 · reddit · 2026-07-21
Benchmarks compare speculative decoding methods on Qwen3.6-27B (dense, NVFP4) on a single RTX PRO 6000 Max-Q, using the same client and three restart samples per point across vLLM and SGLang.
Main results
- DFlash is the clear winner: about 3.3x on SGLang and 2.5x on vLLM, reaching 4.6x on math reasoning.
- MTP / NEXTN delivers roughly 2.2x–2.8x, with speedups improving as draft depth increases.
- EAGLE3 peaks around 1.9x at K=3, then flattens out.
- ngram barely helps, only 1.1x–1.3x.
Engineering notes
- EAGLE3 would not load on vLLM for this model because of an hfhub head-dim validator issue; it only worked on SGLang with a patched build.
- DFlash initially crashed on the first token in SGLang because its sampler does a raw matmul on an NVFP4-quantized lmhead; dequantizing the head once fixes it.
- They also had to cap max-running-requests to avoid a mamba/GDN cache OOM.
More from Infra
- AI Infra Meme: While Countries Need TW-Scale Power for Datacenters, People Argue About Lightbulbs — mgostIH · 2026-10-27
- Legora runs 2,000 parallel agent evals on Modal Sandboxes for legal AI — blelbach · 2026-10-27
- Cloud release velocity scores: AWS 95, Azure 90, Google Cloud 82 — DavidLinthicum · 2026-10-27
- Selling a 5090 to scale local inference: one business weighs 5 hardware paths — Anarchaotic · 2026-10-27
- AWS tutorial: fleet-wide edge CV model lifecycle with SageMaker SDK v3, ~$0.07 per run — AWS ML Blog · 2026-10-27
- Qwen3.8-27B hits 40 tok/s on Jetson Thor with INT4 + MTP, accuracy beats FP16 — No-Dragonfly6246 · 2026-10-27