Reality check: 10 self-distillation agent RL methods fail to consistently beat GRPO
zju · hf · 2026-10-15
Background
Self-distillation methods in agentic RL keep proliferating, but existing evaluations often reuse published baseline scores under differing conditions, rarely compare contemporaneous methods directly, and ignore training stability and compute cost—obscuring where reported gains actually come from.
Setup
The authors compare 10 representative methods against GRPO on ALFWorld, ScienceWorld, Search-QA and WebShop across three model sizes: 132 training runs totaling 12,124.4 recorded H800 GPU-hours, with repeated evaluations over 8 rollout seeds for all pretrained backbones and 11 trained methods.
Key findings
- Gains over GRPO are limited in scope; no method consistently outperforms it across settings.
- Strong final performance can mask unstable training.
- Additional training cost does not consistently translate into better performance.
The benchmark aims to support fair comparison and steer future work toward reliable, cost-effective agent training improvements.
More from coding & agent
- Player-1: RL turns VLM agents into GTA V players with zero game-specific training — yshan2u · 2026-10-15
- Agentic engineering vs AI engineering: one builds software with agents, the other builds agents as the product — Al_Grigor · 2026-10-15
- Cursor + Gemini 4 Argon builds a multiplayer 3D capture-the-flag game in one go — FinanceYF5 · 2026-10-15
- Claude Opus 5.5 levels up codegen but its docs and PR prose are 'unusable' — indragie · 2026-10-15
- Gary Marcus flags AI agent fatigue: managing coding agents burns out engineers — GaryMarcus · 2026-10-15
- The thrill of building isn't gone — you're just not stopping to enjoy it — _jaydeepkarale · 2026-10-15