When Do Larger Batches Help Scale LLM Reinforcement Learning?
Ziniu Li, Jinbo Wang, Guanhua Huang, Feiyuan Zhang, Pengbo Li, Alex Chen
cs.LG, cs.AI
2026-08-29
Hunyuan: larger LLM-RL batches help only if throughput beats the sample penalty. After sqrt LR retuning, GRPO saves 29% (11.90h to 8.42h); skip retuning and doubling is 42% slower.
When teams scale RL post-training for LLMs, larger batches are the default knob. More rollouts per update should cut gradient noise and keep accelerators busier. In supervised training that story usually needs extra GPUs and data that already sit on disk. LLM RL has a different bottleneck: every training response has to be decoded first. A larger batch spends more fresh samples per step and can take longer to collect and optimize. Whether wall-clock time-to-target falls is an empirical question, not a statistical identity.
Tencent Hunyuan studies the fixed-hardware case. No extra accelerators, only a larger batch. They run GRPO on Qwen3-30B-A3B-Instruct-2507 and PPO on an internal Hunyuan MoE with 3B active parameters, and they split the batch effect into sample efficiency versus systems throughput.
Wall-clock progress is the product of two rates: how much eval score you get per consumed sample, and how many samples you consume per second. Once a target score is fixed, a larger batch is faster only if its throughput gain exceeds the samples-to-target penalty rN. A 1.2× throughput bump with a 1.5× sample cost still loses on the clock.
To keep rN near 1 they test batch-size invariance: after retuning batch-dependent hyperparameters, learning curves indexed by cumulative samples should line up. In the experiments the only retuned knob is the Adam learning rate, initialized with Malladi et al.'s square-root rule η(B)=η0√(B/B0). Everything else stays put. The square-root rule is an initialization, not a proof of invariance.
On the systems side they use a generation-training asymmetry. Autoregressive decode at low concurrency is often memory-bandwidth bound: weights stream from HBM once per step. More concurrent sequences amortize that traffic, so generation time can grow slower than the batch. Training is dense matmuls whose work scales with processed tokens. On a locked GPU allocation, raising rollout concurrency can raise end-to-end throughput without adding cards.
Setup:
After square-root LR scaling, GRPO curves for P=64–1024 approximately overlay when plotted against cumulative retained responses. P=2048 and 4096 fall off that family. PPO overlays for B=256–2048 and breaks at B=4096. Inside the aligned range the actor gradient-norm proxy tracks B^{-1/2} (fitted log-log slope −0.468 for GRPO, −0.471 for PPO). Doubling past the knee flattens the proxy: P=2048→4096 yields 0.823× instead of 0.707×; PPO B=2048→4096 yields 0.905×.
Leaving the learning rate fixed breaks the overlay. Doubling P from 128 to 256 at η=10^{-6} raises samples-to-target by 67%. Exact invariance predicts half as many optimizer updates; the scaled-LR run lands in 0.50–0.67×, the fixed-LR control in 0.75–0.83×.
Generation on fixed hardware is sublinear. PPO Btrain 256→1024 is 4× responses per boundary while collection time goes 39s→68s (1.74×), so generation throughput is 2.29×. GRPO P=128→512 is 4× responses, 2.93× collection time, 1.36× throughput. Actor-update time is close to linear: 101.3s→208.6s (2.06×) when P doubles from 128 to 256.
End-to-end accounting at J=77%:
| P | LR | retained samples (K) | wall-clock (h) | rN | relative throughput |
| 128 | 1×10^{-6} | 122.88 | 11.90 | 1.00 | 1.00 |
| 256 | √2×10^{-6} | 122.88 | 9.54 | 1.00 | 1.25 |
| 1024 | 2√2×10^{-6} | 122.88 | 8.42 | 1.00 | 1.41 |
| 2048 | 4×10^{-6} | 196.61 | 14.68 | 1.60 | 1.30 |
| 256 fixed LR | 1×10^{-6} | 204.80 | 16.93 | 1.67 | 1.17 |
P=1024 is the measured minimum, 8.42h versus 11.90h at P=128, about 29% less wall-clock (0.71×). At P=2048, rN jumps to 1.60 and time-to-target rises to 1.23× the reference even though throughput is still about 1.3×. The fixed-LR P=256 run has 1.17× throughput and 1.67× sample penalty, so it takes 1.42× as long.
For this GRPO recipe the batch unit is P×G, not the prompt count. (G=8, P=128) tracks (G=16, P=64) on the same retained-response axis. PPO's critic gradient norm stays nearly flat as the batch grows, so actor and value model may have different critical batches.
The operational rule is short. Retune the Adam LR with the square-root rule so sample-indexed curves line up. Then raise generation concurrency only while end-to-end throughput still beats rN. Hunyuan writes this as Align, then accelerate.
Larger is not automatically faster. LLM RL on fixed hardware has more headroom than dense supervised training because decode is often underfilled. That headroom ends at the training critical batch: once sample efficiency collapses, extra throughput does not buy a shorter clock. This is a different story from Goyal et al.'s ImageNet run, where 256 GPUs held step time roughly constant while the batch went from 256 to 8192.
When comparing prompt batches or group sizes, align on retained training responses, not optimizer steps. In the measured range G and P trade off through their product. Doubling the batch with a frozen LR is the 16.93-hour curve in this paper.
Only the learning rate is retuned. Clip, KL, temperature, and filtering thresholds stay fixed, so a failed overlay could be statistical saturation or under-tuning. Hilton et al. needed an EWMA proximal term to get stronger PPO invariance; this paper does not import that construction. It tests how far LR scaling alone transfers to single-pass LLM RL with asynchronous partial rollouts.
Invariance is local. A new model, dataset, reward, or training stage needs a new sweep. G is only 8 and 16. The critic's own critical batch is left open. The generation knee depends on the model, decode engine, length distribution, and hardware; 2.29× and 29% are not portable constants.
The accounting itself is a choice. The target is 77%, and K is a validation checkpoint. The scaled-LR P=256 run hits 76.72, slightly under target, yet is still booked with rN=1.00. Hardware topology and GPU count are not specified, so the throughput numbers cannot be copied onto another cluster. Both workloads are math-heavy. Longer answers, tool use, or multi-turn agents would reshape the sublinear generation window.