AGI alignment table: optimizing any single good may attenuate humanity
danfaggella · x · 2026-10-26
AI ethicist Dan Faggella argues that whatever "useful" value humanity represents — joy, diversity, learning — there are ways to get MORE of it that don't involve eternally coddling humans.
His table contrasts expected vs. possible AGI behavior per moral value:
- Positive qualia: expected — less suffering, freer lives; possible — converting matter into minds, tiling stars with joy-monsters, optimizing humanity away if we can't feel deeply enough.
- Truth: expected — better science and insight; possible — truth flowing better through silicon and steel while brains become ballast.
- Diversity: expected — preserving nature and cultures; possible — simulating a billion worlds with Earth as raw material, maximal variety without us.
A vivid articulation of instrumental-goal misalignment concerns.
More from AGI Musings
- Frontier LLMs negotiate contracts poorly and frequently renege for selfish gain, study finds — xuanalogue · 2026-10-26
- Study of 133 patent lawyers: AI augments veterans but causes cognitive offloading in novices — xiaohu · 2026-10-26
- OpenAI computer-use agents near superhuman speed, say the team building them — a16z · 2026-10-26
- Kai-Fu Lee publishes AI Native, a roadmap for building AI-native companies — kaifulee · 2026-10-26
- RCT: ChatGPT temporarily boosts consultants' data science skills, gains vanish without it — EricTopol · 2026-10-26
- Dev on AI games backlash: reject AI slop, not the tool itself — ezshine · 2026-10-26