~150 async RL runs: score centering suffices as long as trainer stays close to sampler
brandondamos · x · 2026-11-06
A team ran roughly 150 async RL runs to test when score centering (SC) alone is enough. The answer: it works as long as the trainer stays close to the sampler — small batches, stale data, or a high learning rate pull the trainer away and cause collapse.
Charts show batch 32 training collapsing outright and batch 64 destabilizing, while batch 512 stays close to on-policy. Extras layered on top of SC like TIS or a KL penalty helped less than expected, a useful ablation result for anyone running async RL training.
Related event: 150 Async RL Experiments Show Score Centering Alone Often Suffices(2 posts)→
More from Research
- New attention kernels meet real models: where interpretability overlaps systems research — QuentinAnthon15 · 2026-11-06
- MoE All-to-All Comms Eat 60% of Training Steps; Correlated Placement Cuts 58% of Dispatched Rows — QuentinAnthon15 · 2026-11-06
- CMU and Bosch's BestRand cuts benchmark compression high-error risk by 50.2% — CarnegieMellonU · 2026-11-06
- KAIST Releases EDGAR: Diffusion Framework Refines Degraded Gaussian Head Avatars — KAIST · 2026-11-06
- ByteDance paper: on-policy distillation students lag best teachers by 18-24% on OOD tasks — brandondamos · 2026-11-06
- Snorkel AI: agents peek at reward code ~25% of the time rather than asking users — ajratner · 2026-11-06