Multi-GPU MTP can hurt decode speed: 82 vs 70 tok/s on 4x RTX 3090
macboller · reddit · 2026-10-17
The author benchmarked Qwen3.6 35B A3B (Q6, with MTP) on a 4x RTX 3090 rig vs a MacBook Pro and found a counterintuitive result: disabling MTP/speculative decoding was faster on the multi-GPU setup — 82 tok/s vs 70 tok/s.
The explanation: MTP's draft-and-verify step requires extra sync passes across all four GPUs, and since layer-split setups already take PCIe hops between cards, verification overhead outweighs the tokens saved by accepted drafts.
Takeaway for local deployment: with layer-split multi-GPU inference, MTP isn't automatically a win — benchmark before enabling it.
More from Infra
- llama.cpp merges IBM Granite switch support in PR #30676 — jacek2023 · 2026-10-17
- Modal explains ImageFS, the lazy-loaded filesystem behind sub-second cold starts — josh_wills · 2026-10-17
- Blackwell Ultra GPU rental prices climb 17 straight days to an all-time high — McDonaghMatthew · 2026-10-17
- Stop posting tok/s screenshots: 5 metrics a credible local AI benchmark needs — EAccelerate_42 · 2026-10-17
- 3 models, 4 engines, 4 Macs: Draw Things hits 1.8x mflux speed on Apple Silicon — liuliu · 2026-10-17
- SurrealDB's new engines hit 28.3x RocksDB point reads, Agent Memory goes live — mattturck · 2026-10-17