KD in mid-training favors reasoning over factual recall, AI2/UW paper finds; Switch Distillation proposed
LukeZettlemoyer · x · 2026-09-16
Luke Zettlemoyer shared his team's arXiv paper (2609.01532) on how logit-based knowledge distillation (KD) behaves differently across training stages:
- Key finding: forward KL distillation improves both reasoning and factual recall during pre-training (vs. vanilla NTP), but in mid-training it slows factual recall acquisition even as reasoning gains continue.
- Mechanism: teachers are more confident on procedural than knowledge-intensive data, while students acquire low-entropy factual knowledge early — an asymmetry that distorts mid-training distillation.
- Fix: Switch Distillation uses teacher predictive entropy as a lightweight routing signal, distilling only on tokens where the teacher is confident and falling back to cross-entropy elsewhere.
Authors include Pang Wei Koh, Luke Zettlemoyer, and Wen-tau Yih (UW/AI2/Meta). Highly relevant for teams training small models with a mid-training phase.
More from Models
- Claude upgrade-page traffic rebounded to 17.6M in September, near May's 18.6M peak — firstadopter · 2026-11-02
- Open-source GLiNER-PII scrubs 13 identifiers from a discharge note in 550ms on GPU — MaziyarPanahi · 2026-11-02
- OpenAI Dots user says bloat and broken infra drove him to try newcomer Muse — Koala_Confused · 2026-11-02
- Fable 5.5 reportedly one-shots animation in pure code as Anthropic's coding bet pays off — Hesamation · 2026-11-02
- One agent goal has been running for 4 days, burning 30K tokens — CtrlAltDwayne · 2026-11-02
- Daily AI Brief: Gemini 4 Carbon spotted on Arena, DeepSeek open-sources coding agent — testingcatalog · 2026-11-02