ExpDis decouples exploration from optimization in RLVR, beating DAPO on 7 math benchmarks

micahgoldblum · x · 2026-10-13

NYU researchers (Micah Goldblum et al.) publish "Decoupling Exploration from Optimization in RLVR", introducing Exploration-Distillation (ExpDis) to fix a long-standing problem: adding a novelty bonus directly in RL with verifiable rewards degrades general capabilities.

Approach:

Train one or more explorer policies with a novelty bonus for aggressive exploration;
Filter their trajectories for correctness and quality, then distill them into a separate student policy trained without the novelty bonus;
Alternate exploration and optimization over multiple rounds, insulating the student from the explorer's degradation.

Results: Across 7 mathematical reasoning benchmarks and 2 model families, ExpDis outperforms DAPO at the same wall-clock budget, with improved pass@$k$ scaling—producing more diverse correct solutions. Paper (20 pages), code, and checkpoints are open-sourced.

Original post →

ExpDis decouples exploration from optimization in RLVR, beating DAPO on 7 math benchmarks

More from Models