Why Noised Distillation Works: Self-Correction During Pretraining Keeps Generations In-Distribution

DimitrisPapail · x · 2026-11-17

Commenting on the new perturbed-input distillation results, BlackHC offers intuition: autoregressive models make mistakes while generating, and noising teaches them to self-correct during pretraining. His hypothesis: their own generations become less out-of-distribution — the same issue RL fixes — with possible connections to information bottleneck theory.

Original post →

More from Research

Research channel →