Michael Nielsen essay: ASI x-risk disagreements trace to conviction thresholds, alignment focus may be a mistake
michael_nielsen · x · 2026-09-22
Former OpenAI researcher Michael Nielsen published a new essay exploring why experts disagree so sharply about existential risk from ASI—and arguing that focusing on alignment as the primary goal may be a fundamental mistake.
- Key explanation: expert disagreement stems not from differing information but from differing thresholds for conviction and comfort with reasoning based on toy models and heuristic arguments
- The essay argues ASI makes the world vulnerable to a host of new technologies beyond a rogue ASI—even alignment skeptics should be concerned
- Widely discussed in safety circles, with several readers saying they expected to disagree but ended up agreeing with the main point
More from AGI Musings
- Physicist Quits Tenure-Track Job for AI Safety: 'They Installed Escalators on All the Mountains' — matthew_d_green · 2026-09-22
- AI x-risk debate reignited: distraction conspiracy or inconvenient truth? — AaronBergman18 · 2026-09-22
- Alex Epstein calls (P)doom 'fake threat analysis' that only manufactures fear — TinfoilTricorn · 2026-09-22
- Ruxandra Teslo: AI anxiety is about losing meaning and agency, not just jobs — anshulkundaje · 2026-09-22
- From Erdős to Millennium Problems: AI math progress is compounding fast — haider1 · 2026-09-22
- More inference than training machines means writing will soon beat reading — GregoryDiamos · 2026-09-22