Why AI lab researchers who believe in ~10% extinction risk keep working there
sebkrier · x · 2026-09-09
Jasmine Sun maps the three reasons people keep working at AI labs despite believing in 10% extinction risk:
- Techno-determinism: someone will build ASI anyway; better it's me than China/OpenAI.
- Consequentialism: ASI might kill us but might deliver utopia/immortality — a +EV bet.
- Self-interest: fun, money, cool tech with friends; ignoring the macro picture.
Notably, none of these is "hyping risk for marketing" — people believe what they say while compartmentalizing. She argues the public is better served by researchers pushing consensus on specific safety solutions (regulatory or otherwise) rather than vagueposting about extinction.
Quoted Anthropic researcher EvanHub says he earnestly believes AI could kill all humans — >10% within the next decade — and that Anthropic is trying its best but has no clear plan or path to solving superintelligence alignment.
Related event: Why AI Researchers Who Believe in Extinction Risk Stay on the Job(2 posts)→
More from AGI Musings
- Debate: ASI could arrive in 3-10 years, leaving little time to adapt — ronbodkin · 2026-09-10
- Benjamin Bratton and Marek Poliks lecture: how trillions of agents will reconfigure society — tylerjdunn · 2026-09-10
- Full text of 'Superdark Factory' paper now live on Antikythera, detailing the dark stack's four principles — tylerjdunn · 2026-09-10
- Ex-Continue founder publishes 'The Superdark Factory' in MIT Press: 35T agent tokens/month by 2029 — tylerjdunn · 2026-09-10
- Experiment with 100 AI agents: when 9% cheated on math problems, 24% blew the whistle — weballergy · 2026-09-10
- kuza55: framing AI safety only as 'superalignment or pause' fuels panic — kuza55 · 2026-09-10