Group Entropy-Controlled Policy Optimization
Guangran Cheng, Chengqi Lyu, Songyang Gao, Wenwei Zhang, Kai Chen
cs.CL
2026-07-18
GEPO asymmetrically scales GRPO advantages by group entropy, fixing entropy heterogeneity in mixed-domain RL post-training and halting GRPO's mid-run collapse.
GRPO-style reinforcement learning is now a standard recipe for post-training reasoning models. It samples K answers per prompt and normalizes advantages within the group. The trouble: under one base policy, prompts from different domains sit in very different entropy regimes, the paper measures group entropy around 0.49 for physics and 0.81 for general knowledge. GRPO assumes the normalized advantage is unbiased across groups, and that assumption collapses under entropy heterogeneity. When in-group accuracy p approaches zero, the zero-mean constraint squeezes positive advantages up and dilutes negative ones, producing a self-reinforcing cycle: low-entropy groups get sharper, their entropy drops further, and high-entropy groups receive an ever-weaker learning signal. The result is unstable training and sometimes collapse.
GEPO (Group Entropy-Controlled Policy Optimization) asymmetrically scales the advantage by group entropy. Group entropy is defined as the per-token average of log-probabilities across the K answers for a prompt, giving Hg(x). Low and high thresholds come from batch statistics: Hlow = mean - betalowstd and Hhigh = mean + betahighstd, then smoothed with an exponential moving average.
The scaling rule is the core:
Why asymmetric (alphahigh < alphalow)? The authors' reasoning is that low-entropy groups are more sensitive to aggressive intervention and pushing them too hard triggers length collapse, with answers shrinking shorter. The method intervenes only when needed, adds no sampling cost, and is a lightweight extension on top of GRPO. The paper uses alphahigh=0.2, alphalow=0.5, betahigh=0.3, betalow=0.2, gamma=0.01.
Two base models, 13 benchmarks (math, physics, code, multimodal, instruction following), K=16 responses per prompt.
Intern-S1-mini:
| Method | Average |
| Base | 50.9 |
| GRPO | 51.5 |
| AEPO | 51.8 |
| Clip-Cov / KL-Cov | 51.5 / 51.8 |
| GEPO | 54.2 |
Qwen3.5-9B:
| Method | Average |
| Base | 65.0 |
| GRPO | 70.7 |
| AEPO | 67.1 |
| GEPO | 71.2 |
On individual benchmarks, Intern-S1-mini moves IMO-Bench 29.8 to 38.8, GPQA 65.1 to 69.2, MMMU-Pro 55.0 to 59.8. Ablations: dropping low-entropy control falls to 52.6, high-entropy control to 51.3, the asymmetric shaping to 53.0; every piece contributes.
The most telling result is stability: on Qwen3.5-9B, GRPO collapses around step 170 with AIME25 falling from about 85% to about 40%, while GEPO rises smoothly and monotonically with a stable, bounded entropy trajectory, against GRPO's runaway entropy growth and AEPO's heavy oscillation.
For anyone post-training reasoning models, this paper puts its finger on a real problem: under mixed-domain training, GRPO's group normalization is not a fair reference frame across prompt groups of different entropy. The GEPO fix is light, no extra sampling, no change to the backbone, only the advantage scaling, so it drops straight onto an existing GRPO pipeline. It also takes half the engineering pain of training stability off the table: entropy gating keeps low-entropy groups from collapsing too early and high-entropy groups from starving, visibly lowering the risk of collapse.
The paper has no separate Limitations section. From the content, a few points stand out: the method depends on the grouped-sampling structure (K responses per prompt), so it does not apply to single-sample RL; the gain strength differs across models (+2.7 on Intern-S1-mini, more pronounced, but only +0.5 over GRPO on Qwen3.5-9B), so it is not equally effective on every base; entropy thresholds adapt from batch statistics, yet five hyperparameters (alpha, beta, gamma) still need hand-tuning with no sensitivity analysis given; and the 13-benchmark average hides per-task variance, with some tasks barely moving.