NVIDIA's LSPD brings RL tricks to policy distillation, cutting rollouts by 75%
nvidia · hf · 2026-09-29
NVIDIA researchers reinterpret on-policy distillation (OPD) through the lens of RL, linking the reverse-KL objective to KL-regularized policy optimization and introducing Least-Square Policy Distillation (LSPD), which imports optimistic exploration and off-policy data reuse from value-based RL.
Key points:
- The idealized formulation achieves a sharp O(log K) regret bound under online exploration;
- LSPD outperforms existing distillation baselines by an average of +1.59 points (Avg@16) across six math reasoning benchmarks and diverse teacher-student setups;
- Pass@k evaluations up to k=64 show it better preserves policy diversity;
- Its fully off-policy variant matches vanilla OPD using only the first 25% of rollout batches, greatly cutting sampling cost.
More from Research
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- LatamGPT releases 178.4M-document open Latin American corpus on Hugging Face — OmarUFlorez · 2026-11-16
- Open-source TotalSegmentator spots a 42mm dilated aorta on a routine chest CT — MaziyarPanahi · 2026-11-16