On-Policy Self-Distillation: LLMs self-teach with 4-8x token efficiency over GRPO
burny_tech · x · 2026-09-13
Researchers from UCLA, HKU, and Meta Superintelligence Labs introduce On-Policy Self-Distillation, where an LLM conditioned on privileged info (correct answers or reasoning traces) supervises its weaker self via dense per-token feedback.
- Achieves 4-8x token efficiency vs GRPO, outperforming both GRPO and SFT/off-policy distillation
- Motivation: SFT suffers exposure bias; GRPO needs multiple rollouts with sparse, sequence-level rewards that vanish when outcomes are uniform
- Blog, paper, and code released
More from Research
- Proxy Policy Steering adapts frozen VLA models to new tasks at inference time — weichiuma · 2026-09-14
- New research: standard SGD matches AdamW for LLM RL training, with far less memory overhead — zhaoran_wang · 2026-09-14
- RSI work separates practical harness self-improvement from unproven intelligence explosion — arthurcolle · 2026-09-14
- Amazon Proposes Query-Aware Index Pruning to Optimize Retrieval Under Budget Constraints — _reachsumit · 2026-09-14
- New Paper Finds Retrieval Signals Give No Reliable Routing Gain in Adaptive Multimodal RAG — _reachsumit · 2026-09-14
- Google: Graph RAG Cuts API Hallucination Rate from 56.4% to 16.2% in Java-to-Python Migration — _reachsumit · 2026-09-14