Why the case for AI doom isn't convincing: a 2000-word critique of Yudkowsky's new book
binarybits · x · 2026-09-21
Timothy B. Lee's Understanding AI essay systematically critiques Eliezer Yudkowsky and Nate Soares's book If Anyone Builds It, Everyone Dies. Context: Anthropic CEO Dario Amodei calls himself "relatively an optimist" yet sees a "25 percent chance that things go really really badly"; Altman said the worst case is "lights out for all of us." Lee frames the doom argument as resting on three premises:
- Humans are on a path to superhuman AI;
- Such systems will gain great power over the physical world;
- We don't know how to ensure they use that power for good.
Lee thinks outside skeptics focus too much on premise 1 and are too complacent; his critique targets the latter two — how AI acquires and exercises power, and whether alignment is truly unsolvable. He argues the doom case bakes in unproven assumptions and is ultimately unconvincing, making this one of the most representative counter-arguments to AI x-risk thinking.
More from AGI Musings
- Three emergences in AI: scaling, inference-time scaling, and multi-agent scaling — sqcai · 2026-09-21
- Being against AI use is like being against ambition, argues developer — AaronBergman18 · 2026-09-21
- Prove2Me: Claude agents wrote 13M lines of Lean in 11 days to formalize Fermat's Last Theorem — liuzhuang1234 · 2026-09-21
- EA Called a Distinctive Experiment in High-Agency Applied Philosophy with Few Guardrails — jessi_cata · 2026-09-21
- Meta Muse pitch revives the question: will personal AI agents get real permission controls or just one big Allow button? — yi111 · 2026-09-21
- ctjlewis: nothing can be done now to stop AGI, like stopping the sunrise — ctjlewis · 2026-09-21