Microsoft Research’s ReOPD reuses teacher prefixes to distill multi-turn agents offline
MicrosoftResearch · hf · 2026-07-24
ReOPD turns multi-turn distillation into an offline prefix replay problem
Microsoft Research studies on-policy distillation (OPD) for multi-turn agent tasks and proposes Replayed-Prefix On-Policy Distillation (ReOPD).
- Standard online OPD is expensive because each update needs fresh student rollouts and teacher queries.
- ReOPD reuses pre-collected teacher trajectories as replayed prefixes, so the student acts only at selected steps while the teacher provides dense supervision.
- The paper identifies a prefix trap: later prefixes are more on-policy for the student, but can fall into histories where the teacher’s targets become unreliable.
- To handle that tradeoff, ReOPD uses a step-decaying sampling schedule that emphasizes earlier, lower-shift prefixes.
- Across mathematical reasoning with Python and search environments, ReOPD preserves or improves OPD-level accuracy.
- It uses zero tool calls during student training and is reported to be at least 4× faster per rollout than online OPD.
The core idea is to convert expensive agent-environment interaction into a reusable offline resource for scalable distillation.
More from coding & agent
- Rethinking dependencies in the agent era: drop libs that save 20 lines of code — gethackteam · 2026-10-26
- RustGIS hits 2,000 GitHub stars in two weeks with agent-first architecture — giswqs · 2026-10-26
- Vercel Ships 12 New Products at Its 2026 Conference as It Races Rivals for the Agent Entry Point — FinanceYF5 · 2026-10-26
- Build an AI agent from scratch in under 50 lines of pure Python: ReAct loop tutorial — adnan_hashmi · 2026-10-26
- Dense instance segmentation via Muse Wonder + SAM 3.1 on Meta's Model API — nikhilaravi · 2026-10-26
- Security tip: use AI agents to build a NIST CSF knowledge base for incident response — prasanna_says · 2026-10-26