Alignment debate: most goals in the prior are incompatible with human survival
EigenGender · x · 2026-10-29
A fragment of an AI existential-risk discussion outlining key cruxes: almost all goals in the relevant prior are incompatible with human civilization surviving; as AI approaches superintelligence it will increasingly resemble coherent goal-seeking agents rather than incoherent semi-oracle-like LLMs; and standard ML goal-instilling methods don't fix this. The author plans to expand the argument in a tree structure.
More from AGI Musings
- New paper: agentic economies for scientific discovery—allocating scarce experiment capacity — sebkrier · 2026-10-29
- AI audit of 138 economics papers finds 2,035 errors; 97% of directional errors inflate claims — eigenrobot · 2026-10-29
- 'MLEs get replaced first'? Engineer's agent confidently set the wrong env var — vboykis · 2026-10-29
- MIT Media Lab panel asks: can we have both AI agents and human agency? — patpat_mit · 2026-10-29
- PNAS study of 64,000 people in 64 countries: LLMs miss key differences in simulating happiness — patpat_mit · 2026-10-29
- Seroter's daily reading list: Gemini 4 Carbon, state of context engineering, agent-shaped judgment — rseroter · 2026-10-29