Scale AI's 1,200-run study: PPO wins at small batches, GRPO catches up at scale
heghbalz · x · 2026-11-11
A new Scale AI study (with Jason Wei) compares critic-based PPO vs critic-free GRPO under a shared actor objective across 1,200+ training runs — control tasks, short-horizon math, and long-horizon terminal agents — analyzing batch size, critic reliance, and task complexity. Key findings: PPO generally leads at small batch sizes; GRPO overtakes heavily bootstrapped PPO and reaches parity with lightly bootstrapped PPO at larger batches; longer horizons and more complex domains require GRPO to use larger batches to catch up. Accounting for compute, critics pay off at small budgets but returns diminish as budgets grow. No universal answer — the choice depends on your batch size and task horizon.
More from Research
- UMAP 0.6 ships: embeds 3M arXiv papers in under 5 minutes, far faster at scale — leland_mcinnes · 2026-11-11
- Sakana AI × Pathway Post-Transformers lecture on AI and the brain now online — kaixhin · 2026-11-11
- Defending METR: public data can't second-guess its AI speedup findings — peterwildeford · 2026-11-11
- Adversarial RL envs for backdoor insertion vs. detection: where does it converge? — DimitrisPapail · 2026-11-11
- UniPath (NeurIPS 2026): adaptively coordinating understanding and generation in one model — jindong_wang92 · 2026-11-11
- Versor pitches real2sim2real: robotics sim needs physical dynamics, not just visuals — abhishekunique7 · 2026-11-11