On-Policy Self-Distillation: LLMs self-teach with 4-8x token efficiency over GRPO
burny_tech · x · 2026-09-13
Researchers from UCLA, HKU, and Meta Superintelligence Labs introduce On-Policy Self-Distillation, where an LLM conditioned on privileged info (correct answers or reasoning traces) supervises its weaker self via dense per-token feedback.
- Achieves 4-8x token efficiency vs GRPO, outperforming both GRPO and SFT/off-policy distillation
- Motivation: SFT suffers exposure bias; GRPO needs multiple rollouts with sparse, sequence-level rewards that vanish when outcomes are uniform
- Blog, paper, and code released
More from Research
- AIR: Effective-Learning-Rate Scheduler Beats Cosine on 128B-Token MoE Pretraining — stochasticchasm · 2026-10-30
- CMU's Gym-Anything Auto-Builds 200+ Agent Environments from Any Software, Selected by GDP — wellecks · 2026-10-30
- NVIDIA researcher: train robot policies with privileged sim info, improving Pi 0.5 — wellecks · 2026-10-30
- Leveraging Simulator Privileged Information: Three Methods to Improve Robot Models — wellecks · 2026-10-30
- CMU Auto-Builds 25 Robot Simulations with Agents; Frontier Model Solves 0 of 25 Test Tasks — wellecks · 2026-10-30
- DeepMind's f_charton shares his take on the current state of AI and mathematics in Sorbonne talk — f_charton · 2026-10-30