Michael Nielsen: the real ASI danger is raw power, not a rogue misaligned system

michael_nielsen · x · 2026-11-19

Michael Nielsen resurfaces his April 2025 talk essay "ASI existential risk: reconsidering alignment as a goal", examining why experts disagree so sharply on ASI xrisk — citing Hinton's call to stop development, Bengio's 20% catastrophe estimate, and Altman's pre-OpenAI essay.

His core claim: the fundamental danger isn't a rogue ASI escaping control, but the raw power ASI confers and lowered barriers to dangerous technologies. Two underappreciated consequences: many find rogue-ASI implausible and wrongly dismiss xrisk entirely; and much alignment work, however well-intentioned, accelerates catastrophic capabilities without addressing the world's vulnerability to dangerous tech. Dan Robinson amplified it: "Explained well."

Original post →

More from AGI Musings

AGI Musings channel →