Speculative Self-Distillation cuts training tokens 38–69% with no accuracy loss
Azaliamirh · x · 2026-11-13
Stanford researchers introduce Speculative Self-Distillation (SSD, NeurIPS 2026). Since only a few tokens in each rollout actually depend on new knowledge, a teacher (the same model with extra information) co-writes rollouts—like a tutor taking over only when the student drifts. Results: 38–69% fewer training tokens than on-policy self-distillation at matching accuracy, 130 min vs 414 min to reach on-policy accuracy on Science Q&A, and less forgetting in stress tests (−1.4 vs −4.5 points).
More from Research
- François Fleuret: why isn't EMA normalization more popular than batch norm? — francoisfleuret · 2026-11-13
- Miles v0.2 Debuts at SGLang Summit: RL Training Across NVIDIA, AMD, TPU, Trainium — ying11231 · 2026-11-13
- Open robot learning book adds planning chapter covering CEM and loop control — GlenBerseth · 2026-11-13
- A 14-minute video traces one ChatGPT question from the Enter key to the GPU and back — Glass-Garden-5888 · 2026-11-13
- Reddit proposal: train a corrective LoRA for FLUX Klein from Krea-corrected outputs — Friendly-Fig-6015 · 2026-11-13
- Burkov's weekly: 94% of UK students use AI for assessed work, CUDA Rust, e-means — burkov · 2026-11-13