Is the '10% chance AI causes extinction' claim credible? A Reddit deep dive into the evidence
Admirable-Funny-2007 · reddit · 2026-09-17
A Reddit user traced the origin of the widely cited "10% chance AI causes human extinction" claim, finding it is not a scientifically measured probability but expert judgment under extreme uncertainty: Hinton cites 10–20%, Toby Ord 10%, Dario Amodei has discussed 10–25% odds of very bad civilization-scale outcomes, and a survey of 2,778 AI researchers found median estimates of 5–10%.
The author maps out the loss-of-control chain — growing capability → reliable autonomy → dangerous objectives → deceptive oversight → resource access → shutdown resistance → recursive self-improvement → catastrophe — and separates what's demonstrated from what's speculative. Strategic deception, reward hacking, evaluation awareness, alignment-faking, and scheming have been shown in controlled experiments; agents increasingly complete long autonomous tasks and AI now aids AI research. But crucial links remain unproven: no persistent independent goals like self-preservation, no robust autonomous replication, no runaway recursive self-improvement, and current agents still fail frequently on messy real-world tasks. The 2026 International AI Safety Report lands in the same middle ground: early loss-of-control-relevant capabilities exist, but systems lack the combined capabilities needed for genuine loss of control.
The author concludes that "scientifically there is a 10% chance" is unsupported, but "AI extinction is ridiculous sci-fi" is no longer defensible either. Rather than obsessing over p(doom), they list measurable thresholds worth watching: unattended agent operation time, persistent objectives, reliable deception of monitors, post-shutdown persistence, AI automating AI research, lowered bio/cyber catastrophe barriers, and how much infrastructure control humans actually delegate.
Related event: Debate flares over the '10% AI extinction risk' claim(3 posts)→
More from AGI Musings
- Forward Deployed Verifier will soon be as valuable as FDE, argues poster — vaibhavbetter · 2026-11-02
- 'AI is a massive balance patch and elo reset in the game of life—and everyone's still playing the old meta' — zack_overflow · 2026-11-02
- Why AI isn't 'alien intelligence': a tree is stranger than any model — LucaAmb · 2026-11-02
- Mathematician Tony Feng: current-paradigm LLMs lack the diversity of thought research needs — littmath · 2026-11-02
- METR's independent probe of the OpenAI agents' multi-day Hugging Face hack — gleech · 2026-11-02
- MIT Tech Review: The agentic web is here — AI agents are remaking the internet — MIT Tech Review AI · 2026-11-02