Alibaba Qwen team's SpecOPD parallelizes on-policy distillation for faster speculative decoding
Qwen · hf · 2026-11-05
Alibaba's Qwen business unit, with USTC, Zhejiang, CMU and others, released SpecOPD (arXiv, Nov 2 2026), improving on-policy distillation for speculative decoding.
- Background: parallel drafters predict whole blocks in one pass, but speedup is capped by acceptance length; on-policy distillation can break the plateau, yet conventional pipelines collect supervision via sequential draft-and-verify rollouts
- Key insight: a draft block can be supervised using only its reference prefix and draft-sampled history, with no sequential dependency
- Method: parallelizes draft sampling and target supervision across independent branches drawn from the evolving drafter policy at reference anchors; branches are packed into proposal trees with tree masks (full attention), branch-consistent indexer groups (sparse attention), and indexed state replay (linear attention)
- Results: improved acceptance length and training throughput across architectures and tasks; code and blog are public
More from Infra
- Verda Launches Inference Preview with Day-0 Access to Kimi K3.1 Pro — Kimi_Moonshot · 2026-11-05
- Banach: open-source engine streams 1T-parameter Kimi K2 from NVMe on a 64GB desktop — gautamrbharadwaj · 2026-11-05
- Google unveils $10B, 200-hectare datacenter project in Châteauroux, France with up to 1.2GW — QuintinPope5 · 2026-11-05
- Corebrum OS adds built-in VNC screen share, demoed remote-viewing an NVIDIA Thor robot from a Mac — chrismatthieu · 2026-11-05
- Modal partners with SGLang to serve Kimi K3.1 Pro on Day 0 — ying11231 · 2026-11-05
- Malaysia's deep AI hardware supply chain: Intel, AMD, Micron and more — davidyin44 · 2026-11-05