NYU's ExpDis decouples exploration from optimization in RLVR, beating DAPO

Researcher Micah Goldblum has released a new reinforcement learning framework, Exploration-Distillation (ExpDis), designed to unlock aggressive exploration in LLMs during the RL stage, with performance reportedly surpassing standard algorithms like DAPO by a wide margin. What makes this work notable is that it directly tackles a core tension of the RLVR era: how to encourage exploration without sacrificing the general capabilities a model has already acquired through pretraining.

Confirmed

• The author points out that under RLVR (reinforcement learning with verifiable rewards), LLMs are already pretrained, and the reward only supervises a tiny fraction of the model's knowledge — a model could forget most of Wikipedia and still earn high rewards — so the traditional RL assumption that "high reward equals a good model" no longer holds
• ExpDis's core idea is to decouple exploration from optimization: an RL explorer model explores aggressively, and a stronger student model is then distilled from it
• The author claims ExpDis prevents the degradation of general capabilities and knowledge observed when directly using a novelty bonus, because the degradation of the student model is isolated from that of the explorer
• Given matched training time, ExpDis outperforms RND
• The author reports that ExpDis substantially outperforms standard algorithms such as DAPO

Why it matters

• If the capability loss caused by aggressive exploration can indeed be isolated through a decoupled architecture, RL training will no longer require choosing between exploration gains and general-capability losses — a directly relevant insight for LLM post-training under the RLVR paradigm
• Its explicit treatment of the observation that "rewards cover only a minuscule slice of knowledge" addresses industry concerns about RL training causing models to forget what they know

2026-10-13 ~ 2026-10-13 · 6 related posts

• [source] New RL Framework Exploration-Distillation Boosts LLM Exploration, Outperforms DAPO — micahgoldblum · 2026-10-13
• Researcher Warns Aggressive Exploration in RLVR Can Degrade LLMs — micahgoldblum · 2026-10-13
• Exploration-Distillation Prevents Capability Degradation in RL, Beats RND at Matched Wall-Clock — micahgoldblum · 2026-10-13
• [source] ExpDis decouples exploration from optimization in RLVR, beating DAPO on 7 math benchmarks — micahgoldblum · 2026-10-13

2 near-duplicate retellings: micahgoldblum · micahgoldblum