Bit-exact speculative decoding hits 106 t/s on Qwen3.8-27B with one RTX PRO 6000
stargate425 · reddit · 2026-10-12
A r/LocalLLaMA user details a full recipe for bit-exact speculative decoding of Qwen3.8-27B on a single RTX PRO 6000 Blackwell, reaching 106.6 t/s on short prompts with deterministic inference.
Key findings
- vLLM nightly with DFlash2 k=9 + XQA is fast (110 t/s) but not exact: 15/24 prompts differ from no-speculation output; surprisingly, --max-num-seqs 1 vs 32 alone flips 21/24 outputs.
- SGLang's deterministic mode alone isn't enough — two verify kernel fixes (GDN recurrence tiling order + fp32 beta; verify attention run as decode rows) yield 24/24 exact matches on token ids and logprobs.
- DeepGEMM 0.2.0's sm120 bf16 kernel reads weights at 1/3 DRAM bandwidth at small batch; the author's PR #42145 routes those GEMMs to a Triton tile, bringing exact non-speculative decode to 27.7 t/s.
Performance (single stream, temp 0)
- Short prompts: 106.6 t/s, only 3% behind vLLM
- 2.2k tokens: 73.9 t/s, ahead of vLLM
- 128k tokens: 42.6 t/s, 24% behind; 128k prefill 68.5s vs 44.3s
The author needs bit-exactness to benchmark quants and engines fairly. The post includes a complete recipe: pinned model commits, a gist with all three patches, the DeepGEMM fix PR, and the full launch command.
More from Infra
- WSJ: Inside Silicon Valley's compute wars — Google paid $2.7B for Shazeer, then lost him to OpenAI — dotey · 2026-10-12
- Qualcomm AI uses entropy and CKA to pick layers, beating EAGLE3 defaults in speculative decoding — Qualcomm-AI-Research · 2026-10-12
- Inference is a permanent reverse auction: endless supply and easy switching keep grinding prices down — joecole · 2026-10-12
- 'The most important chart for the AI economy': models commoditize, compute wins — FinanceYF5 · 2026-10-12
- Micro TECs for ELS optical modules could be a $200-600M market by 2028 — jwt0625 · 2026-10-12
- Yandex bot traffic drops 99% after data center attack, Cloudflare Radar shows — rooftopzen · 2026-10-12