Michael Nielsen essay: ASI x-risk disagreements trace to conviction thresholds, alignment focus may be a mistake
michael_nielsen · x · 2026-09-22
Former OpenAI researcher Michael Nielsen published a new essay exploring why experts disagree so sharply about existential risk from ASI—and arguing that focusing on alignment as the primary goal may be a fundamental mistake.
- Key explanation: expert disagreement stems not from differing information but from differing thresholds for conviction and comfort with reasoning based on toy models and heuristic arguments
- The essay argues ASI makes the world vulnerable to a host of new technologies beyond a rogue ASI—even alignment skeptics should be concerned
- Widely discussed in safety circles, with several readers saying they expected to disagree but ended up agreeing with the main point
More from AGI Musings
- OpenAI, Anthropic and DeepMind job posts signal PMs must write evals and code — aakashgupta · 2026-11-12
- PowerShell creator Jeffrey Snover at Ignite: AI revolution is an extinction event for those who don't adapt — dfinke · 2026-11-12
- Worst AI use: signaling you don't care about things you should — Afinetheorem · 2026-11-12
- Stanford HAI blueprint: 56 US agencies report 3,611 AI use cases, evaluation lags — StanfordHAI · 2026-11-12
- Fleuret amplifies debate: AI can't slash wet-lab experiments in unmapped biology — francoisfleuret · 2026-11-12
- Recursive bets on open-endedness: agents running massive parallel experiments like scientist communities — agihouse_org · 2026-11-12