Why Noised Distillation Works: Self-Correction During Pretraining Keeps Generations In-Distribution
DimitrisPapail · x · 2026-11-17
Commenting on the new perturbed-input distillation results, BlackHC offers intuition: autoregressive models make mistakes while generating, and noising teaches them to self-correct during pretraining. His hypothesis: their own generations become less out-of-distribution — the same issue RL fixes — with possible connections to information bottleneck theory.
More from Research
- BrickFlow uses SE(3) flow models to reconstruct buildable LEGO assemblies from images — Peter Kulits · 2026-11-17
- Meta-model writes weight updates from one paragraph, beating LoRA on many benchmarks — gdm3000 · 2026-11-17
- SPARE preprint: 2.55x faster decoding for linear attention on a 1.3B model — krishnan · 2026-11-17
- Study estimates ~12% of philosophy published today may be AI-generated — leland_mcinnes · 2026-11-17
- Berkeley EECS & Statistics open all-rank faculty search in AI for Physical Sciences — berkeley_ai · 2026-11-17
- Origami as physical reservoir computing: powering robotic crawling in Scientific Reports — cephaloform · 2026-11-17