Ambient SFT: Temperature matching against a frozen teacher boosts fine-tuning by up to 21%

giannis_daras · x · 2026-11-19

A new arXiv paper (2611.23229), "Ambient LLMs: Data-Efficient Supervised Fine-tuning via Temperature Matching," by Giannis Daras with Marios Papachristou, Adam Klivans, Antonio Torralba, and Constantinos Daskalakis.

Problem: Post-training (SFT) datasets are far smaller than pretraining data, causing overfitting and catastrophic forgetting of the tails of the data distribution.

Method (Ambient SFT): A frozen copy of the pre-trained model acts as a noisy teacher; the fine-tuned model must match the teacher's conditional distributions at high temperature. The temperature hyperparameter controls trust in the pre-trained model: T→∞ reverts to regular SFT, T=1 becomes knowledge distillation. Learning combines a few clean examples (high variance) with the teacher's noisy predictions (biased), balanced by temperature. The approach draws on Knowledge Distillation and Learning Without Forgetting.

Results: Outperforms standard SFT across domain adaptation, instruction-tuning, and reasoning, with up to 21% average relative improvement. Follow-up work is already underway.

Related event: Ambient SFT: Plug-and-Play Fine-Tuning That Avoids Forgetting(4 posts)→

Original post →

More from Research

Research channel →