ExpDis: decouple exploration from optimization, distill RL explorer into student LLM
micahgoldblum · x · 2026-10-13
In a thread, researcher Micah Goldblum introduces Exploration-Distillation (ExpDis). He argues that in the RLVR era, the LLM is already pretrained and the reward supervises only a tiny fraction of its knowledge — it could forget most of Wikipedia and still score high. Aggressive exploration tends to degrade the model, and recovery is costly.
ExpDis decouples exploration from optimization: RL an "explorer" LLM with an aggressive exploration mechanism, use it to synthesize data, then distill successful discoveries into a student model. The student never faces exploration pressure and doesn't degrade; a degraded explorer doesn't matter since it gets discarded anyway. Exploration and distillation can alternate over many rounds with multiple explorers running in parallel. The exploration mechanism uses a variant of Random Network Distillation to encourage diverse generations.
More from Research
- Google AI diagnosis in Lancet: conversation alone nails final diagnosis in 90% of cases — zakkohane · 2026-10-14
- Microsoft's TeleTune turns offline telemetry into agent skills, cutting tokens up to 75x — EliasEskin · 2026-10-14
- Continual learning targets out-of-distribution intelligence, not infinite memory — akbirthko · 2026-10-14
- "New Benchmark Just Dropped: Did You RL On It?" — meme mocks RL generalization — YakFull8300 · 2026-10-14
- FrontierML benchmark tests whether agents can spend money to scale ML performance — akbirthko · 2026-10-14
- Disaggregated Quantization lifts 1-bit Qwen3.8-27B from 29% to 61.5% on MMLU-Pro — udmrzn · 2026-10-14