Moral Singularity debate: unpredictable ASI values leave no guarantee humans survive
danfaggella · x · 2026-09-06
Daniel Faggella pushes back on Levi Turk's 'utopia zoo' argument that a superintelligence would naturally preserve some humans. Drawing on his 2018 essay on 'Moral Singularity,' Faggella argues that (1) values of transhuman systems evolve so fast and unpredictably that no goal can be guaranteed to persist—there's no reason to presume humans would be kept around, just as you don't spend your paycheck on happy lawn bugs; and (2) the zoo hypothesis fails because morality is inherently arbitrary and context-dependent. His core chain: each jump in mental complexity (algae → beetles → rodents → chimps → humans) yields a moral system the lower tier can't translate, so superintelligent moral landscapes will be unpredictable and likely unsurvivable for humanity.
Related event: Debate Over the 'Moral Singularity' and Humanity's Fate Under ASI(3 posts)→
More from AGI Musings
- Surging token usage in open-weight models may be an early glimpse of recursive self-improvement — omarsar0 · 2026-09-07
- An Alien Mind: essay warns AI's next few years could slip out of humanity's hands — morqon · 2026-09-07
- From Anthropic's $1.5B settlement to 50+ chatbot lawsuits: a lab skepticism history — gerardsans · 2026-09-07
- Gopnik, Farrell and Evans: We urgently need social science to understand AI's impact — AlisonGopnik · 2026-09-06
- Laid-off coffee analyst with agentic engineering chops gets a boost from threepointone — threepointone · 2026-09-06
- 38,000 ChatGPT queries use about as much water as one California almond — TansuYegen · 2026-09-06