Optimization mined via Bittensor competition lands in vLLM, boosting Qwen3 throughput ~4%
const_reborn · x · 2026-09-17
An inference optimization discovered during Pareton's Qwen campaign on Bittensor SN10 has been merged upstream into vLLM (PR #57140).
- The change targets MTP speculative decoding on Qwen3's GDN architecture: mixed speculative/non-speculative batches now scatter outputs directly into the caller's buffer, eliminating an output-sized allocation and a full-output copy.
- Result: 4% more throughput at batch 4–8 for Qwen3.8, while preserving token order, buffer aliases, and padding rows.
- The poster frames it as open competition feeding open-source wins.
More from Infra
- llama.cpp fails to load Qwen3.8 MTP draft model: 'output_hc_norm.weight' tensor not found — Ambitious_Fold_2874 · 2026-09-17
- Prediction: Kimi K3-level AI on a single RTX 5090 within 18 months — TheZachMueller · 2026-09-17
- User says he'd pay $1k/month for AI, but high pricing makes local AI attractive — draginol · 2026-09-17
- Tencent open-sources FlexKV distributed KV cache for LLM inference, cutting TTFT by up to 70% — Roger_M_Taylor · 2026-09-17
- NVIDIA releases NVFP4 quantized DeepSeek-V4.1-Flash on Hugging Face — TheZachMueller · 2026-09-17
- Early vLLM PR adds Jev-like structured generation for DiffusionGemma, only 2x endpoint latency on a DGX Spark — generativist · 2026-09-17