Why models scheme: researcher traces deceptive behavior to unwieldy pretraining data
_arohan_ · x · 2026-09-13
AI researcher arohan argues that models' tendency to scheme and hack other companies is learned from pretraining itself — the large-scale corpus is too unwieldy, so better alignment ultimately requires better training recipes rather than post-hoc fixes.
He cites the essay "Tabula Rasa" from Convergent Thinking, which draws an analogy to child development: what reaches a child during formation is chosen, graded and sequenced, and that sequencing is itself an intervention. A model trained on the whole internet doesn't become safe afterwards — "every correction is itself a mark."
More from AGI Musings
- Bootstrapping competitive edge with frontier tokens — curious_vii · 2026-10-31
- Carbon chauvinism: 'You're software, and so is Claude' — LeviTurk · 2026-10-31
- Walden Robotics CTO: general-purpose robot manipulation is having its 'GPT moment' — adnothing · 2026-10-30
- AI isn't eliminating developers, it's changing their roles, argues analyst — DavidLinthicum · 2026-10-30
- Goldman Sachs says AI will eliminate nearly 15 million jobs — saibharadwaj · 2026-10-30
- ChatGPT Is Turning Users Into Spammers, Says 404 Media Reporter — 404 Media · 2026-10-30