Speculative decoding boosts Qwen3.6-27B on one 5090, but slows crowded servers
luke_pacman · reddit · 2026-07-21
A detailed benchmark of Unsloth’s Qwen3.6-27B NVFP4 on one and two RTX 5090s shows that speculative decoding can help a lot — but only when the server is not already busy.
Key results:
- On 1× RTX 5090, decode speed improved from 66 tok/s to 120 tok/s with nspec=3.
- On 2× RTX 5090, the same setup only reached 108 tok/s, barely ahead of the no-MTP baseline at 99 tok/s.
- With batching, MTP gains collapsed: at 8 concurrent requests it was only +5%, and at 12–16 requests it became a 22–37% slowdown.
- Acceptance stayed roughly stable around 71–73%, suggesting the bottleneck is extra verification work, not rejection rate.
The takeaway is that speculative decoding can be excellent for single-user latency, but once batching and concurrency saturate the GPU, it may hurt throughput instead of helping.
More from Infra
- Arbitrum fee simulation shows higher gas capacity but lower L2 revenue under ArbOS61 — tomwanhh · 2026-07-22
- NVIDIA pushes OpenUSD as the common layer for simulation and physical AI — MonaJalal_ · 2026-07-22
- SkyPilot exits stealth with $20M to unify fragmented GPU compute across five clouds — skypilot_org · 2026-07-22
- Production AI budgets include retries, routing, caching and observability—not just token prices — arx-go · 2026-07-22
- NVIDIA briefs analysts on Vera CPU and doubles down on monolithic agentic design — BenBajarin · 2026-07-22
- NVIDIA unveils Vera Rubin platform with claims of 10x better performance per watt — nvidia · 2026-07-22