Perturbed Inputs Improve Pretraining Distillation, New Paper Finds Gains Even in Multi-Epoch Settings

DimitrisPapail · x · 2026-11-17

A new paper, Perturbing Inputs Improves Pretraining Distillation, makes one change to standard distillation: for the first part of training, both a small student LM and its larger teacher read perturbed versions of the training text (e.g. noise added to token inputs), and the student matches the teacher's next-token predictions on that noisy text before switching back to clean data.

Related event: Perturbing Inputs Boosts Pretraining Distillation, New Paper Shows(3 posts)→

Original post →

More from Research

Research channel →