Salesforce paper: RL-trained experts merged via OPD save 42% FLOPs vs external-teacher distillation
rohanpaul_ai · x · 2026-11-08
A Salesforce AI Research paper systematically compares on-policy distillation (OPD) with GRPO by held-out accuracy per training FLOP across task difficulty, model sizes, and domains. Key findings:
- With an external post-trained teacher, OPD generally loses to GRPO per FLOP and often inflates response length, since every training token requires a teacher forward pass.
- When the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's math accuracy with 50x fewer distillation FLOPs, replicated on code, larger students, and a second model family.
- Across domains, training per-domain GRPO experts then consolidating them via multi-teacher OPD matches joint GRPO on math + code with 42.0% fewer total FLOPs.
Bottom line: to save compute, RL-train domain experts and distill them into one model rather than distilling from a big external teacher.
Related event: Salesforce: On-Policy Distillation Beats GRPO in Compute Efficiency(2 posts)→
More from Research
- MuJoCo 4.0 ships with speculative contacts, permanent soft-body deformation and MuJoCo Studio — ZoubinGhahrama1 · 2026-11-08
- mitsuhiko: djs' JS "heap dumps" may beat real dumps for durable coding agents — mitsuhiko · 2026-11-08
- Albert Gu: Bitter Lesson isn't 'just scale Transformers' — algorithmic innovation matters — sewon__min · 2026-11-08
- Autoware open-sources an off-road driving simulator built on NVIDIA Isaac Sim — rsasaki0109 · 2026-11-08
- The meta-question as agent swarms scale: can models beat gradient descent at its own game? — JensHonack · 2026-11-08
- Why can LLMs solve PhD-level math but still lose at chess? Reddit asks — MesloubFC · 2026-11-08