Perturbing Inputs Boosts Pretraining Distillation, New Paper Shows
A new paper shows that adding input perturbations during pretraining distillation yields significant gains, with student models matching teacher outputs in the early training phase. The simple trick also works in data-limited settings.
2026-11-16 ~ 2026-11-17 · 3 related posts
- New Paper: Adding Input Noise During Pretraining Distillation Beats Standard Distillation — DimitrisPapail · 2026-11-16
- Perturbing inputs improves LM pretraining distillation, new paper shows — DimitrisPapail · 2026-11-17
1 near-duplicate retellings: DimitrisPapail