Microsoft’s ReOPD reuses teacher prefixes to make agent distillation 4× faster

dair_ai · x · 2026-07-27

ReOPD reuses teacher trajectories to cut agent-training cost

Microsoft Research and the University of Amsterdam propose Replayed-Prefix On-Policy Distillation (ReOPD) for agentic tasks.

Instead of fresh environment rollouts on every update, ReOPD reuses pre-collected teacher trajectories as replayed prefixes.
The student acts at selected steps while the teacher provides dense per-step supervision without new environment interaction.
The paper identifies a “prefix trap”: making histories more on-policy for the student can push the teacher onto states where its targets become unreliable.
ReOPD treats this as reliability-aware prefix distribution design and uses a step-decaying sampling schedule that emphasizes earlier, lower-shift prefixes.
Across math reasoning with Python and search environments, it matches or improves accuracy, uses zero tool calls during student training, and runs at least 4× faster per rollout than standard OPD.

More from coding & agent