ByteDance paper: on-policy distillation students lag best teachers by 18-24% on OOD tasks
brandondamos · x · 2026-11-06
A ByteDance-led paper identifies a blind spot in (M)OPD for post-RL consolidation: students achieve near-perfect in-distribution distillation but may not inherit teachers' generalization.
- Across extrapolative settings (math, coding, agents, long-context with Qwen3), OPD/MOPD retain 96-102.73% of teachers' IND scores, yet OOD performance remains 2.45-16.25% below the base model.
- On compositional tasks, OOD performance lags the best teachers by 18.19-24.34%.
- Teacher capability and data coverage are two bottlenecks; jointly alleviating them eliminates OOD forgetting in experiments.
- Proposes geometry-based patching and base-preserving replay to improve OOD performance while preserving IND.
Related event: ByteDance paper reveals generalization limits of on-policy distillation(2 posts)→
More from Research
- UBC and Sony release Solv3R, turning a looped Transformer into a reusable 3D geometry solver — CSProfKGD · 2026-11-06
- Discovery Loop member launches $DISCODEV open AI materials-discovery loop — arkitus · 2026-11-06
- Northwestern turns ordinary smartwatches into hand and gesture sensors with TransfHAR and electrode-based system — realkaranahuja · 2026-11-06
- Three researchers solo-prompting Bing Chat for four weeks gets roasted as a study — Sauers_ · 2026-11-06
- New attention kernels meet real models: where interpretability overlaps systems research — QuentinAnthon15 · 2026-11-06
- MoE All-to-All Comms Eat 60% of Training Steps; Correlated Placement Cuts 58% of Dispatched Rows — QuentinAnthon15 · 2026-11-06