Perturbing Inputs Beats Standard Distillation in Princeton/Microsoft Pretraining Study

udmrzn · x · 2026-11-18

Princeton and Microsoft Research find that knowledge distillation improves when you deliberately corrupt the inputs. Early in pretraining, train the student to match the teacher on perturbed inputs (additive token noise, blended documents, or blended neighboring tokens), then spend the rest of the token budget on ordinary distillation.

Related event: Perturbing Inputs Improves Pretraining Distillation, New Paper Finds(4 posts)→

Original post →

More from Research

Research channel →