Qwen3.6-27B speculative decoding speeds up as quantization gets heavier
thavoc77 · reddit · 2026-07-27
Qwen3.6-27B spec decoding gets better as quantization gets heavier
A Reddit benchmark report says speculative decoding on Qwen3.6-27B improves as quantization gets heavier. Across 10 speculative configurations, the ranking was consistently Q8 > Q6 > Q4 in speed multiplier.
Key findings:
- Acceptance rate was effectively quantization-independent at matched depth.
- The base model step slowed as weight bytes increased, while draft+verify overhead stayed about the same.
- The fastest overall pair was NVFP4 + SGLang.
- DFlash was the fastest algorithm overall.
- Weaver beat DFlash, but it is a fork-only, per-target outlier rather than a general-purpose option.
- MTP was a solid second everywhere.
- EAGLE3 was beaten by both DFlash and MTP, making it mainly a fallback.
- ngram was barely useful, at about 1.03× and negative under concurrency.
- An anomaly: llama.cpp’s MTP path on UD-Q4 was pathologically slow, with Q4 MTP-3 slower than Q6 MTP-3 at the same acceptance.
The author stresses this is a narrow best-case benchmark: greedy, batch 1, short outputs, and one hardware setup. Under concurrency and longer contexts, gains should shrink further. There is also no accuracy A/B yet.
More from Infra
- fmgo: call Apple's on-device Foundation Models from Go with no CGO and no Swift — Super_Run_8466 · 2026-09-23
- Huawei unveils Peerium architecture: nested BSP unifies million processors into one computer — Dr_Singularity · 2026-09-23
- Grok explains why DeepSeek picked DualPipe + ZeRO-1 over ZeRO-3 on 2048 H800s — TheZachMueller · 2026-09-23
- AI costs fall 47% per quarter, 4x faster than DNA sequencing: Epoch AI — daveholtz · 2026-09-23
- M5 Ultra LLM test: 4x faster prompt processing, but double the power draw — DigitalguyCH · 2026-09-23
- $500 of Dell OptiPlexes become a diskless netboot lab where AI agents can't brick the hardware — colinmcnamara · 2026-09-23