Drop the word alignment and the answer is no: scaling AI still risks catastrophe
AndyMasley · x · 2026-09-05
The author reframes the alignment debate with a plain question: if we keep scaling AI models up, will we reliably avoid situations where they do something catastrophic while pursuing a normal goal? He believes the answer right now is no.
He worries many people over-update on what we know about the safe behavior of current models, or assume future models won't have the general "thinking" abilities normal humans have. He adds that the scale of potential catastrophe is bounded by model capabilities — and sees no sign of capabilities stalling out.
Related event: Researcher: Continued AI scaling cannot reliably avert disasters(3 posts)→
More from AGI Musings
- Grady Booch revives Weinberg's 1971 quip: software so buggy one woodpecker could destroy civilization — Grady_Booch · 2026-10-23
- Why Netflix keeps losing to YouTube: UGC platforms professionalize easier than pro platforms go casual — ZeroStateReflex · 2026-10-23
- AI killed the coding moat: indie devs now compete on demand and distribution — sven_ai · 2026-10-23
- DynaVelo, a neural ODE model of single-cell dynamics, published in Nature Biotechnology — anshulkundaje · 2026-10-23
- Beff Jezos: Only frontier labs out-accelerating open source can secure systems — beffjezos · 2026-10-23
- 'I don't want to live in that world': a Redditor on why AGI utopia feels meaningless — peace_venerable · 2026-10-23