Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
Yunpeng Ba, Zhi Zheng, Yue Xie, Jiaqing Li, Xialiang Tong, Tao Zhong, Mingxuan Yuan, Zhichao Lu, Xuyang Wu, Zhenkun Wang
cs.LG
2026-08-28
ES trails GRPO on Pass@1 after GSM8K and DeepScaleR but leads Pass@16/@32 without entropy collapse. Drift is ~40× GRPO; gains sit in sparse large updates.
Evolution Strategies (ES) fine-tune LLM reasoning by perturbing parameters and scoring forward rollouts, with no backpropagation. That saves memory and parallelizes cleanly. ES and Group Relative Policy Optimization (GRPO) both optimize verifier rewards, and they update in different ways. GRPO backpropagates token-level advantages from a group of answers under one policy. ES keeps a population of perturbed policies and averages reward-weighted perturbations into a parameter-space step.
GRPO raises Pass@1 and often collapses policy entropy, so large-K Pass@K can fall below the base model. Separate work has also treated the large parameter drift of ES as catastrophic forgetting. This paper asks three questions: whether ES also narrows reasoning coverage, whether large drift implies forgetting, and which hyperparameters still work as models grow.
ES adds Gaussian parameter noise, rolls out rewards, z-scores them inside the population, and uses those scores as weights on the perturbation directions. The theory writes population diversity as verifier-projected Jensen–Shannon divergence. Parameter noise induces policies with different success probabilities, so sampling across members finds a correct answer more readily than matched repeats from one policy. Reward weighting further tilts mass toward more successful members when weights and success rates correlate. If the center update keeps enough of that margin, Pass@K can rise with Pass@1.
Two settings are used. Easy: Qwen2.5-1.5B/7B-Instruct and Llama-3.2-3B-Instruct, two epochs on GSM8K. Hard: DeepSeek-R1-Distill-Qwen-1.5B, one epoch on DeepScaleR. The comparator is matched-budget GRPO, plus two sequential mixes, ES→GRPO and GRPO→ES. Forgetting checks zero out small-magnitude coordinates and re-evaluate; held-out tasks include GPQA, MBPP, and CSQA. A design sweep covers reward normalization, perturbation scale, population size $N$, and one-point versus two-point estimators.
Pass@K splits cleanly. After GSM8K, six-task averages for Qwen2.5-1.5B are 41.0 / 75.4 / 80.2 for the base (Pass@1 / @16 / @32), 42.9 / 75.1 / 79.9 for GRPO, and 41.5 / 76.0 / 80.9 for ES. Llama-3.2-3B: 44.1 / 74.0 / 78.6 base, 47.1 / 72.7 / 77.0 GRPO, 45.9 / 75.8 / 80.4 ES. On Qwen2.5-7B GRPO still wins Pass@1 (61.0 vs 59.6), while ES leads @16/@32 (80.6 / 83.1 vs 79.0 / 81.5). In the Easy setting GRPO falls below the base on both @16 and @32 in 15 of 18 comparisons. Hard math averages: 47.7 / 73.5 / 77.4 base, 52.9 / 74.7 / 78.0 GRPO, 49.9 / 75.0 / 78.9 ES. ES→GRPO recovers Pass@1 at 52.3 and posts the best @16/@32 (75.8 / 79.2).
During GSM8K, GRPO drops GPQA token entropy; ES keeps it roughly flat.
Geometry: full ES sits 40.7–44.1× farther from initialization than matched GRPO in relative $L2$. At $\tau=1.5\times10^{-3}$, 77.6–93.0% of nonzero updates fall inside that band. Zeroing them leaves target Pass@1 largely intact until sparsity is extreme. The largest ES updates concentrate in LayerNorm and attention projections; GRPO's largest updates sit in embeddings or the LM head, one to two orders of magnitude smaller. Held-out Pass@32 under ES stays at or above the base. Earlier "ES forgets" reports look more like overfitting on a tiny train set.
Hyperparameters: z-score normalization is the ingredient that makes training work. Two-point estimators do not bring the variance win they have in SFT. Needed population shrinks with scale: Qwen2.5-0.5B needs $N=32$ to approach $N=64$, while $N=16$ already suffices at 1.5B and 3B.
ES should not be filed as a weaker, memory-cheap GRPO. It is a different post-training regime: a little less single-sample accuracy, a wider set of successful paths under repeated sampling. If Pass@1 is the production metric, sequential GRPO–ES mixes buy extra Pareto points at the same update budget. Functional sparsity also explains why full-parameter ES still moves billion-scale models: gains do not require every coordinate to change.
In practice, larger models can start with a smaller population, and rewards should be standardized inside the group. If the goal is to surface correct traces that the base policy assigns low probability, ES is a better match than stacking more GRPO steps.
The larger Pass@1 lift still belongs to GRPO; ES alone may undershoot single-sample accuracy in production. Which sequential order wins depends on the task (AIME24 favors ES→GRPO, AIME25 favors GRPO→ES), so there is no stable default. Forgetting claims come from one or two epochs; whether the random-walk component of ES accumulates into real forgetting under multi-task continual training is left open. The theory needs a small center-transfer error for Pass@K gains, and that error is not measured on real LLMs. The LayerNorm concentration is observational; there is no intervention that proves those coordinates are causal.