NVIDIA paper: offline rollouts beat on-policy distillation in 14 of 17 teacher-student pairs
siyan_zhao · x · 2026-10-13
An NVIDIA internship paper challenges the assumption that on-policy sampling is always best for distillation. Across 17 teacher-student pairs, a simple alternative called Semi-OPD — distilling on offline rollouts from the initial student — works better and faster than OPD in 14/17 cases.
Gains correlate strongly with initial teacher-student alignment (Pearson r = -0.73): when poorly aligned, online student rollouts drift away from regions where the teacher gives useful supervision, while initial student rollouts provide a cleaner, more stable signal. Well-aligned pairs (e.g. R1-Distill family) still favor OPD.
Experiments cover Qwen3, R1-Distill, and Nemotron-Cascade pairs, both thinking and non-thinking.
Related event: NVIDIA Study Challenges On-Policy Distillation Superiority(3 posts)→
More from Research
- Anthropic × Adaptyv protein design challenge 3: design a protein to activate GLP1R — victorgreiff · 2026-10-13
- Researcher: Artificial Life will be a key research direction as agentic AI gains self-replication and open-endedness — burny_tech · 2026-10-13
- van der Schaar Lab opens fully funded ML PhD positions for 2027 intake — MihaelaVDS · 2026-10-13
- Scientists identify the 'hangry neuron' in fruit flies that turns hunger into aggression — mjdramstead · 2026-10-13
- LLM cache verifier drift: reject-only calibration tripled error rate; 5% audits fix it — Reasonable_Royal_621 · 2026-10-13
- Sphere Eversion and the Turning Number: Why One Topological Invariant Underlies LLM Embedding Space — ctjlewis · 2026-10-13