Thread: Stronger Models May Survive RL Less Scathed Due to Scale-Invariant Pretraining Priors
1a3orn · x · 2026-09-11
User 1a3orn speculates that intuitions about RL are scrambled because nearness within pretraining priors (emergent misalignment, basic SFT alignment) is scale-invariant for any LLM, being a fact about pretraining data. The user being replied to, voooooogel, notes that holding the RL process roughly constant, more capable models seem to come through relatively less scathed (recent Opus vs Fable 5), since one-shotting a problem means you never have to think about the grader.
Related event: AI researchers debate RL training routes and their side effects(11 posts)→
More from AGI Musings
- Dev warns abliterated models could wreck software supply chains with 1000+ transitive dependencies — nitarshan · 2026-10-27
- Delete your priors: why every SOTA model drop should reset how you think the world works — zack_overflow · 2026-10-27
- Replit predicts every Fortune 500 will be a 'Self-Driving Company' by end of 2030 — amasad · 2026-10-27
- Anthropic says insurers can't cover its own catastrophic risk, X mocks it — GarrisonLovely · 2026-10-27
- MIT's Ziming Liu: AutoResearch doesn't generalize — blind training tricks cause 'forking' disasters — ZimingLiu11 · 2026-10-26
- AI accusations shake literary prizes from Caine to Goncourt: who decides what's human? — TuhinChakr · 2026-10-26