Fukushima analogy for rogue AI: thousands of deaths plausible, extinction far off
sethlazar · x · 2026-09-06
Philosopher Seth Lazar, discussing the magnitude of risk from rogue AIs, drew an analogy to Fukushima: it was a huge international story that changed how power plants are operated and regulated, yet was still 7 to 10 orders of magnitude less serious than human extinction.
He agrees extinction is "outre" and permanent loss of value is very unlikely, but challenges the middle ground: if you allow the mechanism at all, what bounds how bad it could go? Why think rogue AIs could kill thousands but not millions — cause one horrendous industrial accident but not hundreds or thousands? He argues an upper-bound justification is needed.
Related event: Debating rogue AI risk: Fukushima versus Thalidomide(3 posts)→
More from AGI Musings
- "We have not reached AGI yet — 3-5 days for an API integration is insane" — gethackteam · 2026-09-06
- Could RL on Brain Sensors Teach AI Which Tokens Evoke Awe and Disgust? — gabriel1 · 2026-09-06
- OpenAI and Anthropic reportedly sitting on many unpublished internal model results — MoonL88537 · 2026-09-06
- Study: 12.5% AI boosts group consensus by 8%, but at 75% humans adopt the agents' norms — rohanpaul_ai · 2026-09-06
- Zack Korman: AI 'cyber models' won't fix security — patching everything is a convenient myth — gnukeith · 2026-09-06
- Ex-freelance translator turned ML engineer on watching AI automate his old job — mariofilhoml · 2026-09-06