OpenInfer’s Qwen3-4B speculative decoding boosts ShareGPT throughput to 1,288 token/s

青稞AI · wechat · 2026-07-23

This long article explains speculative decoding through the lens of OpenInfer’s implementation on Qwen3-4B.

It shows how speculative decoding can raise throughput on a single RTX 5090, with ShareGPT tests reported at 170 → 381 token/s for single-stream and 576 → 1288 token/s at concurrency 4, while keeping outputs lossless.
The article breaks down why decode is memory-bandwidth bound, why one forward pass normally emits only one token, and how low-entropy spans make multi-token drafting feasible.
It then explains greedy verification, why causal masking preserves exactness, and how sampling complicates the notion of being “lossless.”
Finally, it discusses speculative sampling and the beginnings of draft-model training, including EAGLE-style approaches.

More from Infra