New Paper: Adding Input Noise During Pretraining Distillation Beats Standard Distillation

DimitrisPapail · x · 2026-11-16

A new paper, Perturbing Inputs Improves Pretraining Distillation by Jack Cai with Yihe Dong, Dimitris Papail, and Shivam Garg, tweaks one thing in pretraining distillation: for the first part of training, both the small student LM and the larger teacher read a perturbed version of the text (e.g., noise added to token inputs), with the student matching the teacher's predictions on that input, before switching back to clean text.

Original post →

More from Research

Research channel →