Drop the word alignment and the answer is no: scaling AI still risks catastrophe
AndyMasley · x · 2026-09-05
The author reframes the alignment debate with a plain question: if we keep scaling AI models up, will we reliably avoid situations where they do something catastrophic while pursuing a normal goal? He believes the answer right now is no.
He worries many people over-update on what we know about the safe behavior of current models, or assume future models won't have the general "thinking" abilities normal humans have. He adds that the scale of potential catastrophe is bounded by model capabilities — and sees no sign of capabilities stalling out.
Related event: Researcher: Continued AI scaling cannot reliably avert disasters(3 posts)→
More from AGI Musings
- US State Lawmakers Urge AI Companies to Adopt Verifiable Framework to Slow Development Until Safety Catches Up — EvanHub · 2026-09-05
- Bots beat all humans for the first time in a Metaculus Cup, taking 4 of top 6 spots — NathanpmYoung · 2026-09-05
- Computer use is the fourth exponential demand wave in four years, argues VC firstadopter — firstadopter · 2026-09-05
- Security researchers update on alignment risk after Ajeya Cotra's Dwarkesh interview — Miles_Brundage · 2026-09-05
- Debating a superintelligence ban: critic argues government bans never benefit humanity — AIandDesign · 2026-09-05
- Safety research supply is highly inelastic to money, researcher argues — EigenGender · 2026-09-05