Meta's Switch Distillation Fixes Mid-Training KD Trade-off
A KD study reveals that mid-training forward KL distillation from a post-trained teacher boosts reasoning but harms factual recall. Meta's new Switch Distillation selectively applies logit-based KD based on teacher confidence to avoid the downside.
2026-09-02 ~ 2026-09-03 · 2 related posts
- Mid-Training Distillation Boosts Reasoning But Small Models Lose Fact Recall First — roydanroy · 2026-09-02
- Meta's Switch Distillation: mid-training distillation boosts reasoning, keeps factual recall — meta · 2026-09-03