TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs
Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Wei Lin, Guojun Yin, Ran He
cs.LG, cs.CL
2026-09-27
TGRL converts per-prompt low/high-T reward gaps into JS-weighted token credit, matching a DAPO target up to 36% faster; +1.6% math avg at 32B.
RL with verifiable rewards (RLVR) is how a lot of post-training now improves LLM reasoning: sample completions, score them with a checker, take a policy-gradient step. The bottleneck is exploration under a small rollout budget. The policy collapses into a narrow decoding mode and never finds the higher-reward traces that were sitting one temperature notch away.
The usual fixes either spend more samples (test-time scaling) or turn up temperature, sometimes on a schedule. Extra samples cost budget. Temperature changes diversity, but the training objective never asks whether that diversity raised the verifiable reward on this prompt. Group-relative methods such as GRPO and DAPO normalize inside a same-temperature group. They can rank traces against each other. They cannot estimate whether hotter search beat conservative decoding on that prompt.
If temperature stays a decoding knob, exploration never becomes a gradient.
TGRL, from Meituan with CASIA and UCAS, treats temperature as a within-prompt A/B test and then spreads the resulting signal across tokens that actually move when temperature changes.
The rollout budget stays at 4 responses, matched to the single-temperature baselines:
The gap between the two group-mean rewards is an unbiased estimate of prompt-level exploration gain. Mixed-group GRPO-style normalization injects that gap as an additive shift: if the hot group scored higher, every hot trace is boosted; if not, it is pushed down. The cold traces still enter the mean and variance. They are the ruler, not the update target.
That advantage is still one scalar per trace. For each hot response, TGRL reuses the same logits, builds two next-token distributions by softmax at T0 and T1, and takes Jensen-Shannon divergence (the average KL of each distribution to their midpoint; symmetric and bounded). Large JS means the next-token distribution at that position is the one that jumps when you twist the temperature knob. Weights are log-compressed to tame spikes, then normalized so they average to 1 along the trace, and multiplied onto the hot-trace advantage. Cold-trace token advantages are zeroed. Updating both branches would turn the exploration term into a hot-minus-cold contrastive gradient and pull the policy toward the conservative reference.
A 20-step high-temperature warmup with ordinary group-normalized advantages runs first, so the cold group is not a collapsed baseline. After that, the loss is a clipped importance-ratio surrogate on the hot group only, the same stabilizer PPO uses.
Math is Qwen3 in think mode, trained on DeepScaleR, reported as Avg@16. At 14B, TGRL's six-benchmark average is 69.4 against DAPO's 68.0 (+1.4). AIME24 is 63.4 against the previous best, Dr.GRPO at 61.2 (+2.2). At 32B the average is 70.2 against DAPO's 68.6 (+1.6), first on all six sets, with Olympiad at 69.9 against PPO's 65.7 (+4.2).
| Setup | Metric | TGRL | Strongest baseline |
| Qwen3-14B | six-set math average | 69.4 | DAPO 68.0 |
| Qwen3-32B | six-set math average | 70.2 | DAPO 68.6 |
| Qwen3-4B | LiveCodeBench Pass@16 | 62.1 | PPO / Dr.GRPO 57.7 |
| Qwen3-4B | CodeForces rating | 1574.3 | PPO 1377.6 |
| Qwen2.5-7B | ALFWorld success | 86.7 | PPO 80.4 |
Code is Qwen3-4B on DeepCoder. LiveCodeBench Pass@16 moves from 57.7 to 62.1 (+4.4). CodeForces rating moves from PPO's 1377.6 to 1574.3 (+196.7), percentile 71.9% to 84.3%. On near-saturated HumanEval+, TGRL scores 96.9, behind RLOO at 97.5. At 14B, Minerva (48.9) and AMC23 (92.7) are also not the per-set best.
Agents use Qwen2.5-7B-Instruct. ALFWorld overall success is 86.7 vs PPO's 80.4 (+6.3); Pick2 goes from PPO's 68.8 to 95.0. WebShop success is 74.2 vs EMPG's 69.3 (+4.9). Heat, Cool, and Look are below PPO or EMPG; the overall number is carried by the other subtasks.
Ablations split the two mechanisms. Raising GRPO's temperature from T0 to T1 moves the 14B average 66.5→67.0 and the 32B average 65.1→66.2. Mixed-temperature grouping with uniform token credit reaches 68.2 and 67.8. Adding JS weighting reaches 69.4 and 70.2, with extra +2.5 / +3.5 on AIME25. Entropy-regularized GRPO tops out at 67.8; TAMPO, which schedules one global temperature, at 68.0. Neither catches 14B TGRL.
In Figure 2's Mean@8 curves, high-temperature GRPO lifts AIME24/25 from 54.8%/37.5% to 60.0%/44.2%, still under TGRL's 61.9%/49.6%. Late-stage mean length is 2476 tokens for TGRL vs 3915 (GRPO@T0) and 3482 (GRPO@T1); truncation 7.2% vs 16.0% and 12.4%. On 14B wall-clock, TGRL hits 0.9045 (95% of DAPO's 12-hour best) in 6.39 h; DAPO reaches its own 0.9521 in 10.01 h. That ratio is the paper's claim of up to 36% faster.
This is a drop-in patch for GRPO/DAPO runs whose sample count is already frozen, not a new optimizer. Temperature becomes a within-group contrast. JS reads logits you already have. The appendix complexity argument says the extra term is lower order than the Transformer backward pass; a naive extra forward only changes the constant.
The gains show up on hard contest math, unsaturated code, and sparse long-horizon agents. They vanish on HumanEval+. You have to sample mixed temperatures and spend 1 of 4 rollouts as a non-updated reference. Code is public.
There is no standalone limitations section. The conclusion flags adaptive multi-temperature grouping, richer credit rules, and more open-ended environments as future work.
The 36% figure compares TGRL's time to 95% of DAPO's 12-hour peak against DAPO's time to its own peak, not time to the same final score. The cold group is one sample, so the variance of ΔR in Lemma 1 does not shrink much; that noise lands in the advantage. Each AIME set has 30 problems; the paper says a 5-seed check keeps the mean gap at least as large as the standard deviation, without printing those deviations in the main text. Math and code stay inside Qwen3; agents switch to Qwen2.5-7B. No non-Qwen backbone.
The JS theory says temperature-sensitive positions get at least as much credit as uniform weights would give them. It does not say those positions are the causal bottleneck. In the local-cooling probe, JS spans flip 12.1% of successes and drop score by 0.241, versus 7.6% / 0.152 for an entropy selector. That supports a correlation with downstream reward, not optimality of the credit rule.