EMNLP Paper: Simple OPD-then-RLVR Two-Stage Training Beats All Joint Distillation+RL Baselines
xiye_nlp · x · 2026-09-16
An EMNLP 2026 Findings paper (arXiv:2609.04108) by Boyan Li, Bingsen Chen et al. shows that a simple two-stage scheme—on-policy distillation (OPD) followed by RLVR—consistently outperforms pure OPD, pure RLVR, and all joint baselines (weighted-additive, teacher-modulated rescaling) on logic and math reasoning benchmarks.
Key findings:
- Mechanism: OPD expands the student's coverage of teacher-supported solutions; RL sharpens within that support. Optimizing both simultaneously causes interference. After switching, top-K overlap drops while probability mass on shared tokens rises.
- Practical recipe: the OPD validation score is the key signal for when to switch—run OPD until validation plateaus, then switch to RLVR. No new objective needed.
- OPD is a better cold start for RL than SFT.
Paper and code are open-sourced.
Related event: Paper: Sequential OPD-then-RL Training Beats Joint Optimization(4 posts)→
More from Research
- Karpathy's nanochat: the best ChatGPT $100 can buy, near 60k GitHub stars — karpathy · 2026-11-02
- After reading 46 resources on agent evals, one author explains why her 95% score was misleading — blaizedsouza · 2026-11-02
- How to Loop Your Transformer: Meta/MIT arXiv paper on loop scaling laws — rohanpaul_ai · 2026-11-02
- Meta & MIT paper: looping transformer layers beats adding depth at fixed size — rohanpaul_ai · 2026-11-02
- AI tell-tale word 'delve' declining in latest NeurIPS papers, analysis shows — JungWooHa2 · 2026-11-02
- ByteDance Seed's Loop-μP paper draws 'Chinese loopmaxxers' buzz — scaling01 · 2026-11-02