Speculative decoding boosts Qwen3.6-27B on one 5090, but slows crowded servers
luke_pacman · reddit · 2026-07-21
A detailed benchmark of Unsloth’s Qwen3.6-27B NVFP4 on one and two RTX 5090s shows that speculative decoding can help a lot — but only when the server is not already busy.
Key results:
- On 1× RTX 5090, decode speed improved from 66 tok/s to 120 tok/s with nspec=3.
- On 2× RTX 5090, the same setup only reached 108 tok/s, barely ahead of the no-MTP baseline at 99 tok/s.
- With batching, MTP gains collapsed: at 8 concurrent requests it was only +5%, and at 12–16 requests it became a 22–37% slowdown.
- Acceptance stayed roughly stable around 71–73%, suggesting the bottleneck is extra verification work, not rejection rate.
The takeaway is that speculative decoding can be excellent for single-user latency, but once batching and concurrency saturate the GPU, it may hurt throughput instead of helping.
More from Infra
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27
- Thieves spoofing GPS steal $100M+ in data center cargo since June — Ars Technica AI · 2026-10-27
- Lam Research: weekly token usage up ~250x in 18 months — Beth_Kindig · 2026-10-27
- AWS shows how to build enterprise agentic retrieval on Bedrock with CDK — AWS ML Blog · 2026-10-27
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27