What RL teaches models depends on the base policy's prior, new paper argues
QuintinPope5 · x · 2026-09-28
Quintin Pope argues P(RL learns a behavior) is proportional to its probability under the base policy's prior, citing the arXiv paper "Demystifying Reinforcement Learning Post-Training of LLMs". Implication: making env hacking harder doesn't just make cheating subtler — it directly reduces the learnability of such behaviors.
The paper (Ai2 et al.) deconstructs RLVR post-training in a controlled setting, finding that RL success hinges on whether the base model already places enough probability mass on the desired behavior; it uses policy entropy to compare pretraining/SFT/RL stages and shows 'spurious rewards' effects depend on the post-training prompt distribution.
Related event: Is default alignment broken? Researchers debate RL-induced misalignment(9 posts)→
More from AGI Musings
- Roundtable asks: will Igbo, with 30M+ speakers, have a place in the AI future? — ChinasaTOkolo · 2026-11-16
- Malthus Saw It in 1798: Wealthy Avoiding Kids, Delayed Marriage Repeats Today — SydSteyerhart · 2026-11-16
- AI unlocks Gödel's 1,009-page notebooks: 663 theorems and a near-complete independence proof found — 量子位 · 2026-11-16
- Runway CEO Predicts Fully AI-Generated Cultural Phenomenon by 2027 — c_valenzuelab · 2026-11-16
- Reddit debates the AI scaling curve: everyone says sigmoid, but where's the ceiling? — Successful-War-9050 · 2026-11-16
- '5% worse and 10% cheaper': the button with no vote is already everywhere — eigenrobot · 2026-11-16