Evolution Strategies Beat GRPO on Reasoning Coverage, and Scale Better
burny_tech · x · 2026-09-21
@ethantsliu argues Evolution Strategies (ES) can beat GRPO on reasoning coverage:
- GRPO improves Pass@1 but suffers entropy collapse and reduced diversity at large Pass@K
- ES avoids memory-heavy backprop: it generates a population of slightly perturbed parameter variants, tests their performance, and updates the central model via reward-weighted averaging, transferring diversity directly into the updated policy
- Crucially, ES scales — as the base LLM grows, the required population size actually decreases
Results (train → test):
- Qwen2.5-1.5B (GSM8K → GPQA): GRPO Pass@16 -1.20, Pass@32 -1.77; ES +2.02, +3.90
- DeepSeek-R1-Distill-1.5B (DeepScaleR → MATH-500): GRPO Pass@1 +2.35, Pass@16 -0.64, Pass@32 -1.15; ES Pass@1 +0.20, Pass@16 +0.48, Pass@32 +2.02
More from Research
- What AI means for mathematicians: seven predictions extrapolated from software — cgarciae88 · 2026-09-21
- Stanford RL method fixes VLA latency, lifting robot success from 42% to 97% with 10 minutes of data — burny_tech · 2026-09-21
- Dev Trains Continual-Learning Model From Scratch on 8GB VRAM Laptop, Now at 530M Params — Another__one · 2026-09-21
- Prove2Me: Claude agents wrote 13M lines of Lean in 11 days to formalize Fermat's Last Theorem — liuzhuang1234 · 2026-09-21
- ICLR Review Policy Raises Concerns: 3 Paper Credits Means Mandatory Reviewing — Striking-Warning9533 · 2026-09-21
- EvoOntology: a self-evolving ontology layer bridges the agent-data gap — RUC-DataLab · 2026-09-21