RL training detail: re-prefilling instead of PipelineRL's cached KV, with batch size framed as a GPU-utilization lever
stochasticchasm · x · 2026-09-22
A discussion of unconventional choices in an RL training setup:
- Instead of PipelineRL-style reuse of the old KV cache, the system re-prefills after policy updates. The author speculates that with runs of only 30 steps, the policy shifts enough per step to make re-prefilling worthwhile.
- The choices are otherwise standard, but the large batch size is explicitly motivated as improving GPU utilization more than any training effect — an interesting framing in itself.
Related event: RL training cost debate: grader compute and KV cache trade-offs(2 posts)→
More from Infra
- PyroDash cuts inference cost 96% by having a 4B model call the big one only when needed — jiqizhixin · 2026-09-22
- Underdog's Husky Inference Engine Claims 4.5x Speedup Over MLX, 730 tok/s on MacBook — jimmykoppel · 2026-09-22
- Full-Parameter RL on TPUs: peano_ai Runs 310B MiMo-V2.6 Across 1,000+ TPUs — simonguozirui · 2026-09-22
- Cloudflare Python Workers go generally available after two-year preview — Simon Willison · 2026-09-22
- Fighting AI crawler traffic: beyond Turnstile, Cloudflare's AI Labyrinth as an option — fforres · 2026-09-22
- Software moats won't survive RSI — ML infra's value is demand aggregation, says cHHillee — PatrickToulme · 2026-09-22