Speculative decoding boosts Qwen3.6-27B on one 5090, but slows crowded servers
luke_pacman · reddit · 2026-07-21
A detailed benchmark of Unsloth’s Qwen3.6-27B NVFP4 on one and two RTX 5090s shows that speculative decoding can help a lot — but only when the server is not already busy.
Key results:
- On 1× RTX 5090, decode speed improved from 66 tok/s to 120 tok/s with nspec=3.
- On 2× RTX 5090, the same setup only reached 108 tok/s, barely ahead of the no-MTP baseline at 99 tok/s.
- With batching, MTP gains collapsed: at 8 concurrent requests it was only +5%, and at 12–16 requests it became a 22–37% slowdown.
- Acceptance stayed roughly stable around 71–73%, suggesting the bottleneck is extra verification work, not rejection rate.
The takeaway is that speculative decoding can be excellent for single-user latency, but once batching and concurrency saturate the GPU, it may hurt throughput instead of helping.
More from Infra
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11
- PyTorch Day Korea 2026 launches first offline conf, CFP closes Sept 13 — PyTorch · 2026-09-11
- Local LLM server dilemma: 4x CMP-170HX (price up 53% in 20 days) vs Mac Studio M5 Ultra — rumboll · 2026-09-11
- llama.cpp lands Flash Attention tuning for RDNA4, big prefill gains on AMD — pmttyji · 2026-09-11