Alignment cruxes: almost no prior goals are compatible with human survival
EigenGender · x · 2026-10-29
Researcher EigenGender attempts to articulate the cruxes behind a prominent AI-safety argument, suggesting the debate rests on technical intuitions that haven't been successfully communicated:
- Almost all goals in the relevant prior are incompatible with human civilization surviving;
- As AI approaches superintelligence it will increasingly look like the coherent goal-seeking type for which this is a problem, rather than today's incoherent semi-oracle LLMs;
- Standard ML methods for instilling goals into LLMs are "shallow" and don't generalize;
- Automated alignment researchers won't be good enough to meaningfully help below the capability level where these problems arise.
He concludes the argument's core rests on complex intuitions that even diligent readers haven't been able to extract.
Related event: AI Safety Community Debates the Core Cruxes of Catastrophic-Risk Arguments(3 posts)→
More from AGI Musings
- New paper: agentic economies for scientific discovery—allocating scarce experiment capacity — sebkrier · 2026-10-29
- AI audit of 138 economics papers finds 2,035 errors; 97% of directional errors inflate claims — eigenrobot · 2026-10-29
- 'MLEs get replaced first'? Engineer's agent confidently set the wrong env var — vboykis · 2026-10-29
- MIT Media Lab panel asks: can we have both AI agents and human agency? — patpat_mit · 2026-10-29
- PNAS study of 64,000 people in 64 countries: LLMs miss key differences in simulating happiness — patpat_mit · 2026-10-29
- Seroter's daily reading list: Gemini 4 Carbon, state of context engineering, agent-shaped judgment — rseroter · 2026-10-29