Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards
Fanchao Chen, Hengyu Fu, Shivaram Venkataraman, Jiantao Jiao
cs.LG
2026-09-29
HDL branches where hindsight shifts token likelihoods, reuses prefixes, cuts 35-61% of GRPO tokens, and gains up to 12.5 points on ScienceWorld.
Group-relative RLVR, GRPO and its cousins, learns from reward gaps among several trajectories for the same prompt. The sampling unit is still a full trajectory, so generation dominates cost once traces get long or the task is an agent loop. Systems such as AReaL decouple generation from optimization and can resume partial rollouts. They do not change that unit.
Tokens inside a trajectory are not equal for learning. Wang et al. showed that updating only the 20% highest-entropy tokens can match or beat full-token updates on math. That saves the backward pass, not the generate. TreeRL and BPO reuse prefixes and branch at high-entropy states, without using the observed outcome to reassess earlier choices. PivoARL and R3L ask the model to name retry points, and add extra objectives to teach that skill. The remaining question is how to spend the generate budget: finish a few full traces, let verifier feedback mark where the policy later changes its mind, and resample only from there.
Hindsight-Divergence Localization (HDL) fills a group of G=16 with M=2 complete roots and 14 continuations grown from selected cuts.
For each root, a verifier produces feedback f and the rollout policy writes a reflection r. The pair h=[f; r] is the hindsight context. The same frozen policy is teacher-forced twice over the recorded tokens: once given only the problem and prefix, once given h as well. Each sampled token gets two log-likelihoods. The hindsight-divergence score is the absolute gap. Hindsight can raise likelihoods at key steps of a success and lower them at the error of a failure; the absolute value keeps both. The two highest-scoring positions on each root receive 3 and 4 continuations.
Figure 1 makes the cut concrete. A primality helper accepts 1 as prime and scores 0.7 on the tests. The reflection flags a missing x<2 guard. The largest likelihood shift lands on for, where the root enters trial division. Continuations keep the function header, insert different guards before the loop, and reach reward 1.0. Those suffixes are sampled under the original task context. Hindsight picks the cut; it does not steer generation.
The group uses the same Dr. GRPO objective: mean-centered advantages, no group-std normalization. Roots contribute loss on every generated token. Continuations contribute only on the new suffix, so the shared prefix is not counted again.
The 2x2 layout is load-bearing. On Qwen3-8B Agent, dumping all seven continuations onto one point (2x1) scores 71.09 and trims wall-clock 15%. Growing four roots (4x2) scores 67.87, 4.09 points below the default 71.96, and uses more tokens (0.31M vs 0.27M per step). Several revisits per root, with several siblings at each cut, beat sampling more full traces.
Shared training setup: 128 problems per step, G=16, 200 steps, lr 1e-6, temperature 1.0, slime on four nodes of four GB200 GPUs, asymmetric clip 0.2/0.28, no KL. Models: Qwen3-4B, Qwen3-8B, Llama-3.1-Nemotron-Nano-8B-v1. Math is 4,555 filtered DeepMath-103K items; code 4,063; agent is ScienceWorld with 1,856 task-variation pairs and a 30-action cap. Qwen3 uses thinking mode on math and code, non-thinking on the agent; reflections are always non-thinking. Math and code are avg@N (16 on AIME/HMMT, 8 on Minerva and LiveCodeBench, 4 on OlympiadBench). Agent scores average four episodes per held-out variation. Each method reports mean and std over the three checkpoints with the highest in-domain average.
Versus GRPO, HDL cuts generated tokens 35-61% and rollout wall-clock 18-45%. On Qwen3-8B Math, tokens per step fall from 27.28M to 10.59M. Versus DAPO, tokens drop 43-75% and wall-clock is 1.52x-3.18x faster. Token counts include HDL reflections and DAPO's discarded extra samples.
| Setting | HDL | GRPO | DAPO |
| Qwen3-8B Math (6-bench avg) | 54.16 | 53.17 | 53.47 |
| Qwen3-4B Math avg | 51.86 | 51.57 | 52.48 |
| Llama3.1-8B Math avg | 51.39 | 51.49 | 53.34 |
| Qwen3-8B Code LCB avg@8 | 55.56 | 55.42 | 54.88 |
| Qwen3-4B Agent ScienceWorld | 67.44 | 57.76 | 60.84 |
| Qwen3-8B Agent | 71.96 | 59.50 | 60.80 |
| Llama3.1-8B Agent | 71.81 | 69.94 | 75.35 |
Math and code move by about a point or less. The agent gap is the result that matters: +12.46 over GRPO and +11.16 over DAPO on Qwen3-8B, +9.68 over GRPO on Qwen3-4B. On Llama, DAPO leads Agent at 75.35 and Math at 53.34.
Swapping only the localization signal on the same branching machine, HDL wins all three domains on Qwen3-8B. Agent scores: HDL 71.96, Entropy 66.29, Reflection 65.42. Entropy parks 80.8% of cuts on opening verbs in successful roots and 83.7% in failed ones. HDL puts 52.4% of failed-root cuts on arguments (32.7% when the root succeeds), so continuations can change object or destination while keeping the verb. From steps 1-50 to 151-200, entropy at Entropy's selected positions falls from 1.03 to 0.57 nats and verb TVD from 0.278 to 0.246, tracking frequency. HDL's mean score rises from 6.85 to 7.78. Reflection often returns unparseable or tightly clustered indices, averaging 3.22 valid points per group against HDL's 3.95 (cap 4).
For teams already running GRPO, this is a generation-side patch: same group size, same objective, 35-61% fewer tokens. Long-horizon agents are the fit. Independent resampling replays early mistakes (wrong room, wrong tool) from scratch; HDL spends extra samples where the policy later changes its mind. Do not expect a math or code leaderboard jump. Those lifts sit around 1 point, and some Llama cells lose to GRPO.
Wall-clock already includes reflection and the extra forward pass, and still shows 1.22x-1.83x over GRPO. Do not fix the layout by sampling more roots: 4x2 is slower and 4.09 points worse on the agent setting they ablated.
The extracted paper has no Limitations section. The conclusion claims gains in all three domains. The tables do not. Llama Math is 51.39 vs GRPO 51.49; Llama Code is 54.34 vs 54.87. DAPO wins Llama Agent 75.35 to 71.81. Most of the headline lives in Qwen3 on ScienceWorld, a 30-step text world with dense subgoal reward. Sparse, longer agents and real tool use are untested.
Entropy and Reflection share HDL's root-continuation machinery; they are not full TreeRL, BPO, or PivoARL. The win is the selection criterion. Reflections are generated in non-thinking mode, so the signal quality tracks whether the model can write a useful post-hoc explanation. Thin verifier feedback would turn the score into noise. Runs stop at 200 steps and 8B-class models. There is no measurement of how a bad cut distorts group advantages through the shared prefix.