SFT learns arithmetic mixture, OPD learns geometric mean: why distillation forgets differently

QuanquanGu · x · 2026-10-15

• Quoting researcher @diqiwei's thread: Online Preference Distillation (OPD) forgets less than SFT but breaks under teacher mismatch and on long rollouts.
• The explanation: they optimize different targets. With multiple teachers, SFT (forward KL) learns their arithmetic mixture, while OPD (reverse KL) learns their normalized geometric mean.
• The post also highlights the paper's AI use statement: GPT 5.6 Sol, GPT 6 Astra, Claude Fable 5, and Grok 4.6 helped draft the manuscript and prove several key technical lemmas, with authors verifying all proofs and taking full responsibility.

Original post →

More from Models

• Enterprise AI risk indicator up 9.3x in 2 years, SaaSletter says it leads Anthropic's revenue — thedealdirector · 2026-10-15
• Unverified: Anthropic rumored to be training models to play video games — imjustnewatai · 2026-10-15
• Maya-2 debuts as open-weight Indic voice model, ranks #2 on Hindi TTS leaderboard — CurieuxExplorer · 2026-10-15
• Models may detect AI-generated text by fingerprint alone, and eval sandboxes could be compromised — scaling01 · 2026-10-15
• Third model weighs in: Astra's measured take on automated art vs Fable and Argon — keenanisalive · 2026-10-15
• One prompt, three models: Fable vs Argon disagree on whether artists want automated art — keenanisalive · 2026-10-15

Models channel →