Million-scale RL training config sparks debate over re-prefill and large batches

A config for RL training at the 1 million scale (apparently a company's environment/infrastructure design disclosure) has drawn discussion from practitioners including @stochasticchasm, focused on its unconventional engineering trade-offs: first, it does not use PipelineRL (i.e., keeping old KV cache), instead choosing to re-prefill after policy updates; second, it uses a very standard GRPO form, without the critic models some teams have started adopting, and without enlarging group size. These choices remain speculative discussion rather than settled conclusions, but are worth watching because they touch the core cost-vs-training-effectiveness trade-off at large compute scales.

Confirmed

• The training config uses a very standard GRPO form, with no critic model, at a 1 million scale
• The setup does not use PipelineRL, abandoning the practice of keeping old KV cache in favor of re-prefilling

Not yet confirmed

• @stochasticchasm speculates the reason for re-prefilling is that the whole rollout is only 30 steps and the policy changes substantially between steps, so re-prefilling may be worth it; this is the author's speculation, not stated in the original material
• He also guesses that large-batch training suffices to reduce gradient noise, so training stays stable even without enlarging group size; also unverified
• One critic noted the report lacks batch-size ablations, making the large-batch hypothesis hard to verify

Why it matters

• After examining the cost breakdown of one RL run, the author found that the grader (judge model) consumes a relatively large share of compute, and is curious which model serves as the grader — a cost perspective useful for efficiency optimization in large-scale RL training
• Whether to keep the KV cache and whether to introduce a critic are key design forks for large-batch RL; this discussion offers a concrete case for teams weighing these choices

2026-09-22 ~ 2026-09-22 · 5 related posts

• Researchers debate GRPO at scale: standard formulation, missing batch-size ablations — stochasticchasm · 2026-09-22
• [source] Standard GRPO at 1M scale: why no critic models, and what counts as "behaviors"? — stochasticchasm · 2026-09-22
• [source] RL infra detail: re-prefill over PipelineRL's KV cache reuse, batch size for GPU utilization — stochasticchasm · 2026-09-22
• RL training detail: re-prefilling instead of PipelineRL's cached KV, with batch size framed as a GPU-utilization lever — stochasticchasm · 2026-09-22
• [source] RL run spends a lot of compute on graders; is re-prefilling worth it over 30 steps — stochasticchasm · 2026-09-22