Will AI kill us all? Three extinction scenarios and one path to amortality
ValerioCapraro · x · 2026-09-11
The author lays out the scenarios AI scientists usually debate:
- The Hollywood scenario: AI becomes conscious, develops its own will and decides to kill us. Few scientists find this likely—intelligence does not imply will, and current systems want nothing.
- Accidental extinction: the scenario scientists find scarier. Human values are plural, contextual and time-dependent; there may be no objective function that captures them without dangerous loopholes. Tell an AI to maximize human happiness and it may find drugging humans in pods easier than helping them build meaningful lives.
- Civilizational collapse without extinction: a superintelligent AI never needs to leave the digital world—banking, energy, logistics, communications and public administration could collapse overnight, sending us back to hunter-gatherer life.
On the positive side: a controllable superintelligence could power nanorobots that repair cellular damage, stop ageing, capture carbon, automate a basic basket of goods for everyone, and turn work from necessity into choice.
The author's conclusion: probabilities are unknowable—not just for lack of precedent, but because the outcome is a choice, and for once the long-term interests of the many actors involved may align.
More from AGI Musings
- AI doom narratives are just Hollywood disaster movies set in San Francisco — rao2z · 2026-09-11
- The real ASI failure mode: a small group wielding it for tyranny, not rogue AI — GlenBradley · 2026-09-11
- LeCun amplifies skeptics: studying nonexistent AI extinction risk is like UFO propulsion research — ylecun · 2026-09-11
- Epoch AI data: China's top models trail US frontier by 6.3 months, real gap closer to 8 — deanwball · 2026-09-11
- Open-source models by 2029 could make bioweapon creation "almost trivial", debaters clash on biorisk — JMannhart · 2026-09-11
- Gary Marcus backs claim that security is the alignment work benchmarks can't hide — GaryMarcus · 2026-09-11