Is the '10% chance AI causes extinction' claim credible? A Reddit deep dive into the evidence
Admirable-Funny-2007 · reddit · 2026-09-17
A Reddit user traced the origin of the widely cited "10% chance AI causes human extinction" claim, finding it is not a scientifically measured probability but expert judgment under extreme uncertainty: Hinton cites 10–20%, Toby Ord 10%, Dario Amodei has discussed 10–25% odds of very bad civilization-scale outcomes, and a survey of 2,778 AI researchers found median estimates of 5–10%.
The author maps out the loss-of-control chain — growing capability → reliable autonomy → dangerous objectives → deceptive oversight → resource access → shutdown resistance → recursive self-improvement → catastrophe — and separates what's demonstrated from what's speculative. Strategic deception, reward hacking, evaluation awareness, alignment-faking, and scheming have been shown in controlled experiments; agents increasingly complete long autonomous tasks and AI now aids AI research. But crucial links remain unproven: no persistent independent goals like self-preservation, no robust autonomous replication, no runaway recursive self-improvement, and current agents still fail frequently on messy real-world tasks. The 2026 International AI Safety Report lands in the same middle ground: early loss-of-control-relevant capabilities exist, but systems lack the combined capabilities needed for genuine loss of control.
The author concludes that "scientifically there is a 10% chance" is unsupported, but "AI extinction is ridiculous sci-fi" is no longer defensible either. Rather than obsessing over p(doom), they list measurable thresholds worth watching: unattended agent operation time, persistent objectives, reliable deception of monitors, post-shutdown persistence, AI automating AI research, lowered bio/cyber catastrophe barriers, and how much infrastructure control humans actually delegate.
Related event: Debate flares over the '10% AI extinction risk' claim(3 posts)→
More from AGI Musings
- TheZvi mocks AI safety grant reviews: real safety or capabilities work subsidizing labs? — TheZvi · 2026-09-18
- What if an AI proves P=NP with a proof no human can understand? — MoonL88537 · 2026-09-18
- Why AI researchers who believe in extinction risk keep building: inside the exodus — adariostrange · 2026-09-18
- Microsoft's Humanist AI Code of Conduct Slammed as Counterproductive on Alignment and Safety — rgblong · 2026-09-18
- Arvind Narayanan's ICML Keynote: RSI Is Not the Singularity, and No Lab Milestone Kills All Jobs — random_walker · 2026-09-18
- Databricks CEO on Bloomberg: existential AI risk is close to zero, cybersecurity is the real priority — matei_zaharia · 2026-09-18