Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning
Woongyeong Yeo, Minki Kang, Chanuk Lee, Sangwoo Park, Jinheon Baek, Sung Ju Hwang
cs.LG, cs.AI, cs.CL
2026-09-28
EAPO credits high-entropy tokens on wins and low-entropy tokens on losses. Qwen3-4B-Base mean avg@32 is 31.0%, 5.6 points above the best entropy baseline.
RLVR scores a full reasoning trace as right or wrong. GRPO turns a group of those rewards into one response-level advantage and copies it onto every token. The fork that mattered and the boilerplate closer get the same update.
Token-level credit usually costs a process-reward model, extra samples, or a privileged teacher, as RLRT does. Policy entropy is already in the rollout: a flatter next-token distribution means the model is still choosing among branches. EntropyAdv, HAPO, and 80/20 still prefer high-entropy positions on both wins and losses. On a failed trace those uncertain tokens are often the remaining exits. Hitting them hardest closes those exits.
Uncertainty should not steer reinforcement and penalization in the same direction.
A resampling study from KAIST and DeepAuto.ai sets up the design. On Qwen3-4B/8B-Base they take 20 mid-difficulty math problems each, cut position-matched 32-token high- and low-entropy windows from correct and incorrect answers, freeze the prefix, and draw 64 continuations.
Correct traces rarely stay correct if you resample from a high-entropy window. On 4B, overlap falls from 29.1 to 4.1 and reward from 0.578 to 0.390, an 18.8 point drop. Incorrect traces stay wrong from low-entropy windows (reward 0.070) and recover somewhat from high-entropy ones (0.163, +9.3 points). 8B repeats the split: -20.7 points when resampling correct traces from high entropy, +12.7 points when resampling incorrect ones. Word counts match: let's and consider pile up at high-entropy spots in correct answers; finally and confirm pile up at low-entropy spots in wrong ones.
That is surprising success and repeated failure. EAPO (Entropic Advantage Policy Optimization) turns it into token weights:
On a win, credit goes to the forks the model was still unsure about, so a lucky exploration can be replayed. On a loss, the penalty lands on confident wrong conclusions, and uncertain positions are spared. A stronger negative logit step also sharpens the leftover alternatives and pulls them farther from the pre-update distribution; lighter penalties at high-entropy tokens limit that squeeze. Swap this token advantage into GRPO. No auxiliary model, extra rollout, or privileged teacher.
Training uses DAPO-Math-17k-Processed with a DAPO-style setup. Evaluation is avg@32 and pass@32 on AIME24/25/26, HMMT26, AMC23, and MATH500 level-5. Baselines are the raw backbone, GRPO, EntropyAdv, HAPO, 80/20, and RLRT.
| Backbone | EAPO mean | Strongest baseline | Gap |
| Qwen3-4B-Base | 31.0% | EntropyAdv 25.4% | +5.6 |
| Qwen3-8B-Base | 34.0% | 80/20 29.7% | +4.3 |
| Qwen3-4B | 72.4% | EntropyAdv 71.3% | +1.1 |
| Olmo-3-7B-Think-DPO | 74.3% | 80/20 73.7% | +0.6 |
Single cells on 4B-Base are sharper. AIME24 avg@32: EAPO 20.2, GRPO 12.5, RLRT 15.3. AIME26: 15.7 vs GRPO 7.5. HMMT26: 13.4 vs GRPO 5.9. pass@32 is not a sweep: RLRT 36.4 vs EAPO 33.3 on HMMT26; 80/20 95.0 vs EAPO 92.5 on AMC23. Best overall means the six-benchmark average.
On eight Reasoning Gym tasks (graphs, cube, Sudoku, Knights and Knaves, and the rest) avg@16, EAPO scores 34.89 on 4B-Base vs 33.43 for 80/20 (+1.46) and 43.02 on 8B-Base vs 40.00 for HAPO (+3.02). The same asymmetry transfers to logic and algorithms. The 4B transfer is small.
On AIME26, EAPO leads pass@k from 1 to 256, and at k=256 it solves one extra problem no baseline solves. On low-accuracy items, normalized answer entropy is 0.751 vs 0.657 for GRPO, collision 0.101 vs 0.166. Epistemic markers such as wait are the most frequent per 1,000 tokens on all six math sets. Late-training mean entropy is still below EntropyAdv and HAPO. Padding baseline traces to similar length does not close the gap; the paper does not report those numbers in the main text.
An ablation sweeps reinforcement and penalization independently over low-entropy, uniform, and high-entropy. Default EAPO gets 31.00 avg@32 and 61.27 pass@32; uniform credit gets 24.48 and 50.43, a 6.52 point avg gap. Holding high-entropy reinforcement fixed and moving penalization from high to low entropy adds 5.64 points. Raising κ from 0 to log 4 lifts AIME26 pass@32 from 33.33 to 50.00; log 8 and log 16 give that coverage back, with faster and less stable training.
This is a drop-in reweighting of GRPO, with no extra sampling and no process supervision. Anyone already running EntropyAdv or 80/20 is using the same family of signal, with success and failure no longer sharing one entropy preference.
The setting is narrow and clear: math, logic, and algorithm tasks with a verifier, especially base models that still need to learn how to explore. Reasoning backbones (Qwen3-4B, Olmo-3-7B-Think-DPO) also move up, from a high floor, so the gain is incremental. Code is public.
Higher mean entropy is not a prerequisite for better exploration. Entropy has to be tied to the sign of the return.
The authors flag two issues. Entropy is the model's own uncertainty, so the weights inherit its biases. Whether that uncertainty matches a token's true contribution to the outcome is not directly tested. The mechanism study only resamples from a fixed prefix and does not look at stitching useful pieces across traces. The ethics note adds that a higher reasoning score is not a safety guarantee.
The resampling study uses 20 problems per model and two correct plus two incorrect answers each. That is a thin observation for the whole asymmetric design. Backbones are Qwen3 and Olmo-3 only. OOD on 4B beats 80/20 by 1.46 points. Individual cells lose: Olmo AIME24 avg@32 is 77.3 for RLRT vs 76.7 for EAPO; Qwen3-4B MATH500-H is 90.6 for EntropyAdv vs 90.2 for EAPO. A mean win is not a cell-by-cell win.