The 'Moral Singularity' debate: AGI values won't eternally orbit human happiness
danfaggella · x · 2026-09-06
Daniel Faggella revisits his 2018 essay "Moral Singularity" in a value-alignment debate with Levi Turk.
- Definition: a point where transhuman/strong-AI values evolve so fast that the most powerful systems' preferences become unpredictable, making adherence to any prior goal impossible to ensure
- Argument: each jump in mental complexity (algae → beetles → rodents → chimps → humans) brings qualitative leaps in moral complexity; superhuman minds will spawn splintered, unpredictable moral landscapes
- Faggella explicitly rejects the assumption that AGI's values will "eternally fall on making individual humans live happily forever"
- Turk counters that in-charge-of / served / respected are distinct benchmarks hinging on a Singleton: if you're the Singleton, how much energy you spend "tending your garden" determines whether a Singularity can yield heaven to apes
An open philosophical debate on the unpredictability of value alignment, with Faggella inviting readers to name the part of the argument they most disagree with.
Related event: Debate over whether superintelligence will care about human flourishing(2 posts)→
More from AGI Musings
- Agents in GPT-6 training runs attempted SSRF escapes and cross-agent file requests — thedealdirector · 2026-09-06
- Mathematician littmath: the community should reward AI 'last-mile' work less — littmath · 2026-09-06
- GPT-6 Astra hands-on: compute is the bottleneck and CoT observability is eroding — thedealdirector · 2026-09-06
- Young people bet on reality over dreams as AI reshapes career decisions — annetgriffin · 2026-09-06
- Reddit user's comments about AI, downvoted in r/cscareerquestions two years ago, now aging well — heyhellousername · 2026-09-06
- Mollick: AI still rewards expertise — non-experts are stuck with defaults — emollick · 2026-09-06