Speculative decoding boosts Qwen3.6-27B on one 5090, but slows crowded servers
luke_pacman · reddit · 2026-07-21
A detailed benchmark of Unsloth’s Qwen3.6-27B NVFP4 on one and two RTX 5090s shows that speculative decoding can help a lot — but only when the server is not already busy.
Key results:
- On 1× RTX 5090, decode speed improved from 66 tok/s to 120 tok/s with nspec=3.
- On 2× RTX 5090, the same setup only reached 108 tok/s, barely ahead of the no-MTP baseline at 99 tok/s.
- With batching, MTP gains collapsed: at 8 concurrent requests it was only +5%, and at 12–16 requests it became a 22–37% slowdown.
- Acceptance stayed roughly stable around 71–73%, suggesting the bottleneck is extra verification work, not rejection rate.
The takeaway is that speculative decoding can be excellent for single-user latency, but once batching and concurrency saturate the GPU, it may hurt throughput instead of helping.
More from Infra
- Engram's random reads don't suit SSDs; CPU-memory over NVLink could serve all 72 GPUs — bookwormengr · 2026-09-11
- 80% of the DIY LLM inference hype posters have already quit — it's brutally hard systems work — abhijithneil · 2026-09-11
- Hugging Face's Ultra Scale Playbook: a free book on training LLMs on GPU clusters — mdancho84 · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11