The 'moral singularity' argument: unknowable ASI values rule out 50-50 optimism
danfaggella · x · 2026-09-21
Daniel Faggella and Levi Turk debate ASI risk probabilities. Turk argues that quoting a 12-in-a-trillion chance of 'million+ year planet-scale zoos' implies one actually knows ASI outcomes will be bad; Faggella responds that ASI's effects are unknowable, making a 50-50 'it'll go well' bet logically incompatible with genuine unknowability. His 'moral singularity' essay argues that transhuman AI values will evolve so fast that the most powerful systems' values become unpredictable, and since morality is inherently arbitrary and malleable, AGI will produce splintering moral complexity that humanity may not survive.
Related event: Debate Erupts Over Whether Unknowable ASI Outcomes Mean 50-50 Odds(2 posts)→
More from AGI Musings
- AGI.WTF conference sells 100+ tickets with just five days from announcement to start — NathanpmYoung · 2026-09-21
- Schmidt says UIs will largely disappear as agents generate them on demand; Musk agrees — elonmusk · 2026-09-21
- Yudkowsky: ASI built with current techniques means ruin; P(doom) conflates two probabilities — ESYudkowsky · 2026-09-21
- AI Safety Worker Explains Why He Joined EA: If a Small Group Doesn't Act, No One Will — JacquesThibs · 2026-09-21
- Honglak Lee at Nobel Prize Dialogue: AI accelerates science, but skepticism matters more — honglaklee · 2026-09-21
- Veteran dev: AI built a full MMO in 2 hours, so what am I for? — devontec · 2026-09-21