Fukushima analogy for rogue AI: thousands of deaths plausible, extinction far off
sethlazar · x · 2026-09-06
Philosopher Seth Lazar, discussing the magnitude of risk from rogue AIs, drew an analogy to Fukushima: it was a huge international story that changed how power plants are operated and regulated, yet was still 7 to 10 orders of magnitude less serious than human extinction.
He agrees extinction is "outre" and permanent loss of value is very unlikely, but challenges the middle ground: if you allow the mechanism at all, what bounds how bad it could go? Why think rogue AIs could kill thousands but not millions — cause one horrendous industrial accident but not hundreds or thousands? He argues an upper-bound justification is needed.
Related event: Timothy Lee vs Seth Lazar debate how bad rogue AI can get(20 posts)→
More from AGI Musings
- Grady Booch revives Weinberg's 1971 quip: software so buggy one woodpecker could destroy civilization — Grady_Booch · 2026-10-23
- Why Netflix keeps losing to YouTube: UGC platforms professionalize easier than pro platforms go casual — ZeroStateReflex · 2026-10-23
- AI killed the coding moat: indie devs now compete on demand and distribution — sven_ai · 2026-10-23
- DynaVelo, a neural ODE model of single-cell dynamics, published in Nature Biotechnology — anshulkundaje · 2026-10-23
- Beff Jezos: Only frontier labs out-accelerating open source can secure systems — beffjezos · 2026-10-23
- 'I don't want to live in that world': a Redditor on why AGI utopia feels meaningless — peace_venerable · 2026-10-23