Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
Xianlei Zhou, Xiangdi Meng, Yu He, Tianyu Qi, Shuyan Guan, Xianli Zhang, Jian Zhang, Xin Li, Qika Lin, Jun Liu
EMNLP 2026 main conference
cs.CL
2026-08-24
ERPO replaces Policy-KL with Query-KL on prompt likelihood. On six math benchmarks, Avg@32 rises from GRPO's 0.274 to 0.336, and high-temperature decoding stays much stabler.
LLM policy optimization almost always carries a Policy-KL term: keep the current response distribution close to an SFT reference. Keep the term and the action-side exploration budget gets spent. Drop it and there is no explicit drift control. Stability and exploration are bolted to the same regularizer.
The AMAP group at Alibaba argues the uncontrolled piece sits on the input side. A training corpus can stay fixed while the model's sequence likelihood on each training query still co-evolves with the parameters. They call that quantity the policy-induced query distribution ρθ, and they call its KL to the pre-RL reference ρθ0 environment drift. Figure 1 is blunt. With Policy-KL held to a fixed budget, Query-KL keeps climbing and Policy-KL stays flat. Constraining answers does not stabilize the query process.
ERPO moves the regularizer onto queries. Two pieces.
Proposition 1 is the structural claim. The QKL gradient flows only through ∇θ ℓθ(q). The response score function ∇θ log πθ(o|q) never appears, so QKL puts no direct pressure on the answer distribution. Exploration stays on the action side.
Drop-in for GRPO, PPO, or REINFORCE is three steps: cache ℓθ0, reweight the outer per-query sum, swap Policy-KL for QKL. Inner clipping, baselines, and group-relative advantages stay as they are. Experiments keep the coefficient α at the default 0.01 so the comparison matches the usual Policy-KL strength.
Training uses Qwen2.5-Math-7B and Qwen2.5-32B in EasyR1, on MATH Level 3-5 (about 8.5K problems), 240 steps, eight rollouts per query, 3K token cap. Evaluation covers AIME24/25, AMC, MATH500, Minerva, and OlympiadBench, averaged over temperatures 0.1 to 1.5 so a single decoding temperature cannot hide a collapse.
Six-benchmark means after that temperature sweep:
| Method | Avg@32 | Pass@32 | Pass@1 |
| Base | 0.143 | 0.463 | 0.149 |
| GRPO | 0.274 | 0.575 | 0.275 |
| ERPO | 0.336 | 0.611 | 0.332 |
Versus GRPO that is +6.2, +3.64, and +5.69 points. The largest single-bench jump is MATH500 Avg@32, 0.528 to 0.677, the "up to 14.9%" in the paper. Minerva Pass@32 is the exception: ERPO 0.500, a touch below GRPO's 0.516.
High temperature is the sharper contrast. On MATH500 with Qwen-7B and eight rollouts, Pass@1 at temperature 1.5 is 0.40 for GRPO, identical to the untuned base. ERPO with reweighting reaches 8.60; raising α to 5×10^{-2} reaches 15.00. With 16 rollouts the same temperature is GRPO 10.60 versus ERPO 56.20. On 32B the gap is larger still: at temperature 1.5, GRPO 25.20 versus ERPO 80.80. The low-temperature band (≤1.0) moves from 81.62 to 84.60; the high band (1.2-1.5) from 57.20 to 82.80.
Ablation splits the two pieces. Query-KL alone, no reweighting, takes the Qwen-7B low-temperature mean to 80.90 against GRPO's 68.80, a 12.1-point gap; the paper quotes a 15.9% average lift. Reweighting alone (GRPO) sits at 76.14. Query-KL by itself drives the batch Query-KL from GRPO's 0.9679 down to 0.0041. Full ERPO lands at 0.0828, with Policy-KL still around 0.0728, the same order as GRPO's 0.0601, without an explicit Policy-KL term.
At 1K training steps, GRPO holds for temperatures below 1.0 until about step 240 (15 epochs). After step 400 the high-temperature band drops first, then the collapse spreads to every temperature. ERPO dips as well, and even improves in the high-temperature band for a stretch, but the authors say it is not collapse-proof: entropy can still spike and sampling can still die on long runs. The appendix shows the classic reward-hacking signature for GRPO at step 240: train accuracy still 76.70, Eval@TP1 down from about 75 to 58.40. ERPO's Eval@TP1 at the same step is 78.40. The train-eval gap shrinks from 6.47% to 3.14%, roughly half. Wrapping DAPO with ERPO adds 10.24 points on the ≤1.0 mean; wrapping RLOO adds 2.28. Removing every KL term, training does not converge.
This is a change in where the regularizer sits, not a new advantage estimator. Anyone already running GRPO or PPO can try it with a small patch: cache ℓθ0 once, add QKL, drop Policy-KL. Extra forward cost is essentially zero.
The evaluation protocol is the more useful warning. At temperature 1.0 after 240 steps the two methods are not that far apart. At 1.5, GRPO can fall all the way back to the base model. A single-temperature Pass@1 will hide that failure mode. If serving uses hot sampling or multi-path search, an input-side constraint is worth a run.
The gain is still incremental. Assumption A1 (pinning ρθ near ρθ0 preserves generalization from earlier stages) is a premise, not a theorem, and every number is on math reasoning with Qwen.
The authors list three: math-only and Qwen-only; query-likelihood estimates can vary in quality and cost with data selection and scale; α was not swept, and 0.01 exists to match the old Policy-KL coefficient.
A few more caveats sit in the setup. "Environment" here means the model's likelihood of the prompt. The actual sampling distribution ρtrain never moves. That is an analogy to classic RL transition-kernel drift, not the same object. The weight w(q) ∝ ℓθ0(q) is not an unbiased density ratio, and clipping to 2 makes it less so. On Qwen-7B with eight rollouts, ERPO without reweighting actually wins the ≤1.0 band (80.90 vs 78.74 with weights); the weights mainly help the hot-temperature tail. GRPO never learned the required thinking-tag format, so every reported number is answer accuracy only, with no format-compliance comparison. ERPO also loses on Minerva Pass@32. At 1K steps ERPO can still collapse, just more slowly.