Perturbing inputs improves LM pretraining distillation, new paper shows

DimitrisPapail · x · 2026-11-17

New paper "Perturbing Inputs Improves Pretraining Distillation" (with Yihe Dong, Dimitris Papail, Shivam Garg) changes one thing in pretraining distillation: for the first part of training, both teacher and student read perturbed text (e.g. token-level noise) and the student matches the teacher's next-token predictions on it, then training switches back to clean text.

Related event: Perturbing Inputs Boosts Pretraining Distillation, New Paper Shows(3 posts)→

Original post →

More from Models

Models channel →