Qwen's SpecOPD: branch-parallel on-policy distillation speeds inference up to 3.56x
pmttyji · reddit · 2026-11-06
Qwen released SpecOPD, a branch-parallel on-policy distillation (OPD) framework for improving speculative decoding drafters:
- Key insight: OPD supervision need not inherit the sequential draft-and-verify dependency — a draft block can be supervised using only its reference prefix and draft-sampled history, enabling parallel sampling and supervision across independent branches.
- Engineering: branches are packed into proposal trees to share target-prefix computation; tree masks for full attention, branch-consistent indexer groups for sparse attention, indexed state replay for linear attention; drafters trained with an E2E-TV surrogate directly targeting acceptance length.
- Scale: 8B full-attention (GQA), 27B full-linear (GQA-GDN), and 125B sparse-linear (QSA-GDN) targets; three parallel draft architectures (DFlash, DSpark, DFlash V2); four task domains.
- Results: up to +41.45% mean acceptance length, 16.09× supervised-token throughput over sequential-rollout OPD, and up to 3.56× generation speedup over autoregressive decoding.
Paper, blog, and GitHub code are all open.
Related event: Qwen Open-Sources SpecOPD, Speeding Speculative Decoding Up to 3.56x(2 posts)→
More from Infra
- Batch APIs: the 50% LLM discount only pays off after model choice and prompt caching — arpit_bhayani · 2026-11-13
- RTX Pro 6000 bricks mid-Qwen run, owner jokes it "died of a broken heart" over Jensen news — wine_dark · 2026-11-13
- Bittensor's Chutes Handles 256 Concurrent Requests on a Single RTX 5090 at ~18k tok/s — markjeffrey · 2026-11-13
- Google expands TPU v8 lineup: 8tx cuts latency 2x, 8i Max doubles silicon per rack — Beth_Kindig · 2026-11-13
- Horace He to share two mental models for reasoning about inference at SGLang Summit — cHHillee · 2026-11-13
- Used A100 prices fall another 20-30% as sellers rush ahead of a collapse — Bascilian · 2026-11-13