OpenInfer’s Qwen3-4B speculative decoding boosts ShareGPT throughput to 1,288 token/s

青稞AI · wechat · 2026-07-23

This long article explains speculative decoding through the lens of OpenInfer’s implementation on Qwen3-4B.

• It shows how speculative decoding can raise throughput on a single RTX 5090, with ShareGPT tests reported at 170 → 381 token/s for single-stream and 576 → 1288 token/s at concurrency 4, while keeping outputs lossless.
• The article breaks down why decode is memory-bandwidth bound, why one forward pass normally emits only one token, and how low-entropy spans make multi-token drafting feasible.
• It then explains greedy verification, why causal masking preserves exactness, and how sampling complicates the notion of being “lossless.”
• Finally, it discusses speculative sampling and the beginnings of draft-model training, including EAGLE-style approaches.

Original post →

More from Infra

• NVIDIA says Nemotron 3 Ultra hit 97.1% on agentic RTL chip-design tasks — NVIDIAAI · 2026-07-27
• Local Qwen models power a robot that tests 78 smartphones’ battery life — gappyvalley · 2026-07-27
• MiniBot 2.40 adds xAI, HF Studio and vLLM support with inline media tools — Creative-Type9411 · 2026-07-27
• Apple smart glasses, Nvidia-SK AI data center deal, and Ctrip’s RMB 5.179 billion fine headline a tech roundup — APPSO · 2026-07-27
• DeepSeek funding rumor, EU AI transparency rules and OpenAI agent incident make a packed AI news roundup — 创业邦 · 2026-07-27
• QuixiCore argues native quantized kernels beat dequant-then-generic execution — QuixiAI · 2026-07-27

Infra channel →