Speculative Self-Distillation Cuts Training Tokens 38–69% With Less Forgetting
heghbalz · x · 2026-11-13
A NeurIPS 2026 paper introduces Speculative Self-Distillation (SSD): the teacher only generates the few tokens in a rollout that actually depend on new knowledge, letting the student write the rest. This matches on-policy self-distillation accuracy with 38–69% fewer training tokens, reaches on-policy accuracy in 130 vs 414 minutes on Science Q&A, and forgets less (−1.4 vs −4.5 points in stress tests).
Related event: Speculative Self-Distillation Cuts Training Tokens by Up to 69%(2 posts)→
More from Research
- CSHL benchmark: AlphaGenome 2 leads only within ~100kb of TSS on distal enhancer effects — anshulkundaje · 2026-11-13
- MASCOT (EMNLP 2026) trains multi-agent role-played dialogue, hitting 80.0 persona consistency — jiqizhixin · 2026-11-13
- Stanford's CS312 'Deep Learning Alchemy' opens quizzes and materials to the public — stanfordnlp · 2026-11-13
- Blogger Works Through Stanford CS31: Residuals Matter More Than Normalization — stanfordnlp · 2026-11-13
- Stanford Course Shares Data: AI Users Score Higher on Homework, Worse on Exams — stanfordnlp · 2026-11-13
- Cartwheel Claims Breakthrough in Hand-Object Kinematics Capture from Any Video — andrew_n_carr · 2026-11-13