Why Self-Distillation Beats GRPO and RLHF in Scaling Continual Learning
AI Engineer · youtube · 2026-08-12
In this technical talk, Ronak Malde dives into the challenges of scaling continual learning for long-horizon tasks, such as trajectories with hundreds of tool calls. He notes that on-policy self-distillation often causes models to collapse into hedging, overusing tokens like "wait" and "but" until everything turns into "maybe."
Malde proposes a self-distillation algorithm where the model acts as its own teacher by inserting privileged information (hints) into the teacher's prompt and matching the log probs of a student that never saw the hint. This approach satisfies four key properties—including on-policy sampling and per-token rewards—outperforming GRPO and RLHF by optimizing across the entire vocabulary. The talk also dissects failure modes like hint leakage, the self-distillation analogue of reward hacking.
More from Research
- Visualizing AI Math Progress: Cost to Solve Erdős Problems Nears $10M — ajeya_cotra · 2026-08-12
- ApexFold: Predicting Peptide Secondary Structure Based on Chemical Environment — KevinKaichuang · 2026-08-12
- Study Reveals Shared High-Dimensional Object Spaces in Human and Macaque Visual Cortex — martin_hebart · 2026-08-12
- Physical Intelligence Co-founder: Robotics is Entering its GPT Era — Y Combinator · 2026-08-12
- Engram Talk: Why LLM Training on Private Data Collapses — AI Engineer · 2026-08-12
- Gemma 4 QAT Shows Significant Improvement in KV Cache Quantization, KLD Benchmarks Reveal — Anbeeld · 2026-08-12