MRPD replays expert RL logs to consolidate agents at ~1/7 the cost of MOPD

illinois · hf · 2026-11-20

A new paper proposes Multi-expert Replay Policy Distillation (MRPD): instead of resampling environments like MOPD, it replays experts' RL training logs — rollouts with sampling probabilities and rewards, in the experts' own learning order. This quasi-on-policy replay needs no environment or reward model calls, costs only 1/7 of expert training, and produces students that beat online RL baselines and same-expert MOPD across reasoning, search, and SQL domains. Notably, the order of logs matters more than their amount or rewards — expert training logs act as a curriculum, not a data mixture.

Original post →

More from Research

Research channel →