Ambient SFT: Plug-and-Play Fine-Tuning That Avoids Forgetting
Researchers led by Giannis Daras propose Ambient SFT, a plug-and-play alternative to standard SFT that treats the pre-fine-tuned model as a noisy teacher, using temperature matching to avoid catastrophic forgetting and achieving up to 21% relative improvement on small datasets.
2026-11-19 ~ 2026-11-19 · 4 related posts
- Ambient SFT: treating the pre-finetune model as a noisy teacher, up to 21% gains — giannis_daras · 2026-11-19
- Ambient SFT: temperature matching fixes catastrophic forgetting in fine-tuning, gains +20 IFEval without losing old skills — giannis_daras · 2026-11-19
2 near-duplicate retellings: giannis_daras · giannis_daras