Ambient SFT treats the pre-trained model as a noisy teacher, beating vanilla SFT by up to 21%

giannis_daras · x · 2026-11-19

Giannis Daras introduces Ambient SFT, a drop-in replacement for vanilla SFT that addresses catastrophic forgetting when fine-tuning LLMs on small datasets.

Inspired by Knowledge Distillation and Learning Without Forgetting; follow-ups already underway.

Related event: Ambient SFT: Plug-and-Play Fine-Tuning That Avoids Forgetting(4 posts)→

Original post →

More from Research

Research channel →