RL training detail: re-prefilling instead of PipelineRL's cached KV, with batch size framed as a GPU-utilization lever
stochasticchasm · x · 2026-09-22
A discussion of unconventional choices in an RL training setup:
- Instead of PipelineRL-style reuse of the old KV cache, the system re-prefills after policy updates. The author speculates that with runs of only 30 steps, the policy shifts enough per step to make re-prefilling worthwhile.
- The choices are otherwise standard, but the large batch size is explicitly motivated as improving GPU utilization more than any training effect — an interesting framing in itself.
More from Infra
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- Potion UI's local login system turns your PC into a multi-user AI server — cocktailpeanut · 2026-11-12
- AWS Finally Built Its Own AWS Emulator — super_linear · 2026-11-12
- Megakernel is multiprogramming for GPUs: 'GPUs are entering the CPU age' — pranjalssh · 2026-11-12