Async RL study: score centering alone often matches on-policy, extra TIS/KL fixes don't reliably help
kastnerkyle · x · 2026-11-06
A side-project study tests whether stacking truncated importance sampling (TIS) or KL penalties on top of score centering (SC) helps stable async RL.
- With Qwen3.5-4B on DAPO-Math-17k (staleness 15), SC alone often matched on-policy performance
- When SC fell short, adding more corrections didn't reliably help: vanilla IS collapsed around step 100, SC+TIS around step 300, SC alone lasted to 550
- Takeaway: SC does most of the work, and extra corrections don't rescue it when it fails
More from Research
- Frank Nielsen's taxonomy maps every distance and divergence from Euclidean to quantum geometry — pickover · 2026-11-06
- Anthropic report's key estimates are "guesses" — framing hides massive uncertainty, researcher says — daveholtz · 2026-11-06
- Project AURORA: a big-bet AI Cancer Co-scientist initiative launches with CRUK and NCI backing — zakkohane · 2026-11-06
- PNAS paper: kids explore farther than RL agents, driven by empowerment not info gain — _akpiper · 2026-11-06
- Google open-sources Agile Deliberation, an AI framework that turns fuzzy classification rules into explicit ones — aigclink · 2026-11-06
- ETH RSL's ACP humanoid: reference-free contact planning runs onboard at 50Hz — ChongZzZhang · 2026-11-06