Michael Nielsen essay: ASI's real risk is raw power, not rogue AI
jachiam0 · x · 2026-09-12
Jacques amplifies Michael Nielsen's April 2025 essay "ASI existential risk: reconsidering alignment as a goal," calling it essential reading.
Key points:
- Experts deeply disagree on ASI xrisk: Hinton urges considering halting development, Bengio puts 20% on catastrophe, Altman once called superhuman intelligence the greatest threat.
- Nielsen argues the core danger isn't a "rogue ASI" but the raw power ASI confers, lowering barriers to creating dangerous technologies.
- Two underappreciated consequences: dismissing rogue-AI scenarios leads people to wrongly dismiss xrisk entirely; and much alignment work, while well-intentioned, accelerates catastrophic capabilities without addressing the world's vulnerability.
More from AGI Musings
- Diamandis slams viral ex-Anthropic researcher's 300M-view AI doom tweet as self-destructive hype — rohanpaul_ai · 2026-09-12
- Doomer take: mathematicians have one year before open AI models sweep Millennium Prizes — teortaxesTex · 2026-09-12
- Maybe Cloud Atlas Got the AI Apocalypse Right, Says Thread — eigenhector · 2026-09-12
- The final UI is clear: language in, intelligence, language or generated UI out — signulll · 2026-09-12
- Garry Tan amplifies contrarian take: the 'one-person company' is overrated — garrytan · 2026-09-12
- Fly connectome work framed as the inflection point of AGI and techbio — shakoistsLog · 2026-09-12