Katja Grace: Treating AI doom as a utopia trade-off is a reasoning error, not a plan
KatjaGrace · x · 2026-09-26
AI researcher Katja Grace's essay Doom as a bad method not a utopia trade-off attacks a common reasoning pattern in AI debates:
- Citing her 2023 survey of 800 AI researchers, she notes most assign meaningful probability to both doom and utopia—and many then 'net' the two into a 'net positive on AI' verdict.
- She calls this a wild error, analogous to calling driving 200mph to a new job optimistic because there's 'only' a 10% chance of a fiery death against a 30% chance the job changes your life. The real comparison is the current route to advanced AI versus alternative routes.
- She concedes one can debate whether constraining risky projects risks irrecoverable ruin, but argues most people aren't thinking in trade-off terms for that reason—rather because pros-vs-cons framing is simpler and the topic too abstract to notice the mismatch.
Core claim: accepting some doom risk should be judged as a bad method in itself, not priced in as the cost of a utopia trade.
Related event: Katja Grace: Treating AI Doom as a Path to Utopia Is a Reasoning Error(2 posts)→
More from AGI Musings
- Polymarket puts 16% odds on Anthropic announcing a training pause before November — Polymarket · 2026-09-26
- Add 'can't write professional emails' and the AI x-risk answer flips — louisvarge · 2026-09-26
- Anthropic researcher Carlsmith says AI could be 'justified in going rogue' if mistreated — Polymarket · 2026-09-26
- Superhuman coder, bad emailer: why capability cues skew AI x-risk predictions — louisvarge · 2026-09-26
- Hot Take: AI Is History's Most Powerful 'Joule', 10x Human Energy-to-GDP Efficiency — FinanceYF5 · 2026-09-26
- VC Stewart Alsop: Efficient AI Could Crush Data Center Financing, SF Has Lost Its Edge — StewartalsopIII · 2026-09-26