Michael Nielsen: the real ASI danger is raw power, not a rogue misaligned system
michael_nielsen · x · 2026-11-19
Michael Nielsen resurfaces his April 2025 talk essay "ASI existential risk: reconsidering alignment as a goal", examining why experts disagree so sharply on ASI xrisk — citing Hinton's call to stop development, Bengio's 20% catastrophe estimate, and Altman's pre-OpenAI essay.
His core claim: the fundamental danger isn't a rogue ASI escaping control, but the raw power ASI confers and lowered barriers to dangerous technologies. Two underappreciated consequences: many find rogue-ASI implausible and wrongly dismiss xrisk entirely; and much alignment work, however well-intentioned, accelerates catastrophic capabilities without addressing the world's vulnerability to dangerous tech. Dan Robinson amplified it: "Explained well."
More from AGI Musings
- If AI execs truly believed the risk, they'd be coordinating a stop — they aren't — wfithian · 2026-11-19
- AI Safety Community Emotionally Processing x-Risk: 'My Biggest Regret Is Not Doing This Earlier' — trevposts · 2026-11-19
- "Obsolete ideas have tenure": why AI experts anchor on outdated views — repligate · 2026-11-19
- Stanford researcher: most scientific output doesn't really need peer review — anshulkundaje · 2026-11-19
- Stanford prof Anshul Kundaje: most scientific output doesn't really need peer review — anshulkundaje · 2026-11-19
- Dan Robinson: AI is extremely dangerous even if fully aligned, so Manhattan-project safety plans misfocus — michael_nielsen · 2026-11-19