Stanford's Speculative Self-Distillation cuts training tokens 38-69% with less forgetting

khademinori · x · 2026-11-13

A Stanford AI Lab team introduces Speculative Self-Distillation (SSD, NeurIPS 2026), tackling catastrophic forgetting when teaching an LLM new documents. Student and teacher (the same model with extra information) co-write rollouts: the teacher lets the student write and takes over only when it drifts too far, keeping training as on-policy as possible. Results: 38–69% fewer training tokens than on-policy self-distillation at matched accuracy, 130 vs 414 minutes to on-policy accuracy on Science Q&A, and less forgetting (-1.4 vs -4.5 points) in a stress test.

Original post →

Stanford's Speculative Self-Distillation cuts training tokens 38-69% with less forgetting

More from Research