Sequential Beats Joint: OPD-then-RLVR two-stage training consistently outperforms joint optimization
xiye_nlp · x · 2026-09-16
An EMNLP 2026 Findings paper shows a simple two-stage scheme — on-policy distillation (OPD) followed by RLVR — consistently beats pure OPD, pure RLVR, and all joint baselines (weighted-additive or teacher-modulated) on logic and math benchmarks. The mechanism: OPD expands the student's coverage of teacher-supported solutions, RL sharpens within that support, and jointly optimizing the two signals causes interference. Practical recipe: switch when the OPD validation score plateaus, and OPD is a better RL cold start than SFT. After switching, top-K overlap drops while probability mass on shared tokens rises, showing RL sharpens within the teacher's support.
Related event: Paper: Sequential OPD-then-RL Training Beats Joint Optimization(4 posts)→
More from Research
- GDPx: SemiAnalysis's frontier benchmark across the 11 most valuable domains — rohanpaul_ai · 2026-11-02
- SemiAnalysis launches GDPx: best frontier model scores just 12% on expert work — rohanpaul_ai · 2026-11-02
- Detecting covert instruction-following via internal representations parallels J-space work — dioscuri · 2026-11-02
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Mistral Large 4 writes a quantum chemistry SCF solver from scratch with one short prompt — MistralAI · 2026-11-02
- Hunting for the most concrete "curing cancer with AI" roadmap of 2025 — iskander · 2026-11-02