KD in mid-training favors reasoning over factual recall, AI2/UW paper finds; Switch Distillation proposed
LukeZettlemoyer · x · 2026-09-16
Luke Zettlemoyer shared his team's arXiv paper (2609.01532) on how logit-based knowledge distillation (KD) behaves differently across training stages:
- Key finding: forward KL distillation improves both reasoning and factual recall during pre-training (vs. vanilla NTP), but in mid-training it slows factual recall acquisition even as reasoning gains continue.
- Mechanism: teachers are more confident on procedural than knowledge-intensive data, while students acquire low-entropy factual knowledge early — an asymmetry that distorts mid-training distillation.
- Fix: Switch Distillation uses teacher predictive entropy as a lightweight routing signal, distilling only on tokens where the teacher is confident and falling back to cross-entropy elsewhere.
Authors include Pang Wei Koh, Luke Zettlemoyer, and Wen-tau Yih (UW/AI2/Meta). Highly relevant for teams training small models with a mid-training phase.
More from Models
- Cartesia's new voice model family draws praise; WER alone can't capture context-correct speech — buckymoore · 2026-09-16
- Replication of no-CoT evals shows GPT-Astra makes a qualitative jump across all datasets — dhadfieldmenell · 2026-09-16
- Anthropic's Astra tops spend while OpenAI's Luna dominates token usage by a lot — gdb · 2026-09-16
- DeepSeekMath-V2 makes verification the product, scaling verifier compute ahead of the generator — le_james94 · 2026-09-16
- ChatGPT Plus Work Projects Bug Persists for Days While OpenAI Marks It Resolved — OnwardUpwardForward · 2026-09-16
- Google reportedly building math-specialized Gemini DeepThink Mathematica model — basedjensen · 2026-09-16