RL run spends a lot of compute on graders; is re-prefilling worth it over 30 steps
stochasticchasm · x · 2026-09-22
Noting that a recent RL training run spends comparatively large compute on graders, the author wonders which model was used as grader, and argues that with only 30 total steps the policy shifts significantly each step, so re-prefilling the grader may be worthwhile.
More from Research
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Architecture study sparks monitoring concerns over a multi-hop planning spike — ChowdhuryNeil · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12