The six-step power-seeking AI x-risk argument, broken down for credence calibration
AndyMasley · x · 2026-11-07
Andy Masley shares Joe Carlsmith's canonical articulation of the power-seeking AI x-risk story, suggesting you assign credence to each step to get your footing in the debate. The six-step chain (by 2070):
- It becomes feasible to build "APS" systems: Advanced capability (outperforming top humans at research/strategy/engineering/persuasion), Planning, and Strategic awareness
- Strong incentives exist to build and deploy APS systems
- Building APS systems that would NOT seek power in unintended ways is much harder than building ones that would, yet remain superficially attractive to deploy
- Some deployed APS systems will seek power in unintended, high-impact ways (> $1 trillion aggregate damage) due to objective problems
- This power-seeking scales to permanently disempowering all of humanity
- That disempowerment constitutes an existential catastrophe
Each premise can be independently credited or doubted — that's the framework's value.
More from AGI Musings
- Musk: agentic internet traffic already exceeds human traffic, Cloudflare's 1000x forecast is an underestimate — elonmusk · 2026-11-07
- Healthcare ransomware attacks up 6x since 2018 as AI agents close the patch gap, a16z says — luisdans · 2026-11-07
- US productivity grew 2.1% in the AI boom — only a quarter to half of dot-com peak — emilygsands · 2026-11-07
- Kelsey Piper: Pangram's loudest critics are writers secretly using AI themselves — TuhinChakr · 2026-11-07
- Interpretability ranks lowest in ICLR 2027 review scores, sparking AI-generated-paper hypothesis — nsaphra · 2026-11-07
- Public shaming of AI labs works: $75B redirected, gag clauses scrapped, Anthropic backpedaled — KatjaGrace · 2026-11-07