MRPD replays expert RL logs to consolidate agents at ~1/7 the cost of MOPD
illinois · hf · 2026-11-20
A new paper proposes Multi-expert Replay Policy Distillation (MRPD): instead of resampling environments like MOPD, it replays experts' RL training logs — rollouts with sampling probabilities and rewards, in the experts' own learning order. This quasi-on-policy replay needs no environment or reward model calls, costs only 1/7 of expert training, and produces students that beat online RL baselines and same-expert MOPD across reasoning, search, and SQL domains. Notably, the order of logs matters more than their amount or rewards — expert training logs act as a curriculum, not a data mixture.
More from Research
- Beaker autonomously finds 100+ grading errors across five popular AI benchmarks — VictorSanh · 2026-11-20
- Stanford-led review maps deep learning for gene regulation across cell states — anshulkundaje · 2026-11-20
- GPT-6-Astra Scores Far Below Published Results When Robot Benchmarks Aren't Tailored — wzenus · 2026-11-20
- Cornell mathematicians on 'Big Math': algorithms now decide loans and admissions — and we no longer understand them — stevenstrogatz · 2026-11-20
- Developer claims he 'solved JEPA': model 30x smaller, 100x faster, trains in 30min on a 3080 — MarcJSchmidt · 2026-11-20
- Precise 3D localization of real-world captures unlocks change detection, AR/VR and robotics — bilawalsidhu · 2026-11-20