RL training shifts behavior regimes across scales, not pretraining priors
1a3orn · x · 2026-09-11
A researcher argues that intuitions about RL are often scrambled: alignment or misalignment within pretraining priors (e.g. "emergent misalignment") is scale-invariant because it's a fact about pretraining data. In RL, however, specific circuits are amplified much more sparsely — as papers on RL sparsity suggest — which is scale-dependent, so models go through different behavior regimes at different scales, no longer mediated by pretraining data nearness.
Related event: AI researchers debate RL training routes and their side effects(11 posts)→
More from AGI Musings
- CJR: journalists should stop hyping AI's supposed humanity — just say 'rogue' — ShakeelHashim · 2026-10-27
- PhD applicant emails up 4x YoY due to AI — writing without AI now makes you stand out — AI4Code · 2026-10-27
- Weil's collapse shows BigLaw dies from eroding belief, not a broken business model — jkubicki · 2026-10-27
- All 8 Transformer paper authors left Google, and rival labs captured most of the frontier value — signulll · 2026-10-27
- Anthropic's S-1 pledges AI for curing diseases and serving 4.6B underserved patients — petergyang · 2026-10-27
- Goldman finds statistically significant negative link between AI adoption and US employment growth — ben_j_todd · 2026-10-27