arXiv paper unpacks OPD vs GRPO: self-teacher distillation cuts FLOPs 50x, expert merging saves 42%
rohanpaul_ai · x · 2026-11-08
The arXiv paper "Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO" (Salesforce AI Research) systematically analyzes when on-policy distillation beats RL on compute:
- With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP (up to 7.1x worse), even when the teacher itself is GRPO-trained, and suffers response-length inflation and degenerate behavior.
- The picture flips when the teacher is the student's own GRPO checkpoint: OPD recovers that checkpoint's math accuracy with 50x fewer distillation FLOPs (135.6x fewer than with an external 8B teacher), consistently across larger students, code, and a second model family.
- Across domains, multi-teacher OPD consolidating per-domain GRPO experts costs only 5.2%–8.0% of specialist training FLOPs and reaches joint-GRPO performance with 42% fewer total FLOPs.
Related event: Salesforce: On-Policy Distillation Beats GRPO in Compute Efficiency(2 posts)→
More from Research
- Subliminal Learning Debate: Implicit Prompt Learned as Global Steering Vector, Not Specialized Rules — thebasepoint · 2026-11-08
- Author clarifies perturbation analyses offer many lessons, not one — ajratner · 2026-11-08
- Human2Any Transfers Human Video Skills to Robots with Zero Real-World Training Data — danfei_xu · 2026-11-08
- spearfish Paper: Δ-Mapping Speeds Up Long-Read Alignment to Redundant Transcriptomes — anshulkundaje · 2026-11-08
- "Intelligence Is Not a Scalar": An X Thread Argues Architecture Research Still Matters for AGI — max_paperclips · 2026-11-08
- nanoGPTJAX Author Plans Hardware-Specific Kernels and an NVFP4 Showcase — A_K_Nain · 2026-11-08