NVIDIA: Offline Student Rollouts Often Beat On-Policy Distillation in 14 of 17 Pairs
nvidia · hf · 2026-10-12
An NVIDIA paper questions whether on-policy distillation (OPD) is always needed, proposing Semi-OPD, which distills on offline rollouts from the initial student.
- Across 17 teacher-student pairs from 1.5B to 235B parameters, Semi-OPD wins in 14, with up to +13.6% accuracy and 11.4x training speedup.
- An output-token overlap ratio quantifies teacher-student alignment: OPD helps only when overlap is high. For misaligned pairs, student rollouts drift increasingly off-policy w.r.t. the teacher as context grows, weakening the distillation signal.
- Semi-OPD distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens, making it more stable.
More from Research
- METR's AI time-horizon chart hasn't updated since May, leaving frontier model gains unmeasured — haider1 · 2026-10-12
- 15 detectors, 3 benchmarks: best prompt-injection detector catches 2% of real agent injections — lulzxdxdxd · 2026-10-12
- HANDBOOK.md benchmark: top agents pass just 36.2% of 65 policy-heavy agentic tasks — bibryam · 2026-10-12
- AI finds solutions hiding in plain sight across 22 scientific fields, Science reports — Spare-Dingo-531 · 2026-10-12
- Terence Tao blog: what mathematicians should know about Lean's reliability and AI — thomasahle · 2026-10-12
- Michael Levin publishes a starter pack for his lab's work on diverse intelligence — drmichaellevin · 2026-10-12