Microsoft Research’s ReOPD reuses teacher prefixes to distill multi-turn agents offline

MicrosoftResearch · hf · 2026-07-24

ReOPD turns multi-turn distillation into an offline prefix replay problem

Microsoft Research studies on-policy distillation (OPD) for multi-turn agent tasks and proposes Replayed-Prefix On-Policy Distillation (ReOPD).

Standard online OPD is expensive because each update needs fresh student rollouts and teacher queries.
ReOPD reuses pre-collected teacher trajectories as replayed prefixes, so the student acts only at selected steps while the teacher provides dense supervision.
The paper identifies a prefix trap: later prefixes are more on-policy for the student, but can fall into histories where the teacher’s targets become unreliable.
To handle that tradeoff, ReOPD uses a step-decaying sampling schedule that emphasizes earlier, lower-shift prefixes.
Across mathematical reasoning with Python and search environments, ReOPD preserves or improves OPD-level accuracy.
It uses zero tool calls during student training and is reported to be at least 4× faster per rollout than online OPD.

The core idea is to convert expensive agent-environment interaction into a reusable offline resource for scalable distillation.

More from coding & agent