The Moral Singularity: unpredictable AGI values bode poorly for humanity
danfaggella · x · 2026-09-23
Daniel Faggella argues 'uncertainty' shouldn't be equated with safety, coining 'Moral Singularity': a point where transhuman/strong-AI values evolve so fast that we can't predict what the most powerful systems will value — making it impossible to ensure adherence to any prior goal. Morality is arbitrary and contextual; escalating mental complexity (algae → beetles → rodents → chimps → humans) suggests AGI will birth splintered, unpredictable moral complexity that humanity may not survive.
Related event: Daniel Faggella's 'Moral Singularity': AGI Value Drift Is Unpredictable(2 posts)→
More from AGI Musings
- OpenAI's economics team: 'We don't have the nouns yet' for the jobs AI will create — paulnovosad · 2026-09-23
- AI engineering is more like lawmaking than board games, argues Drew Breunig — dbreunig · 2026-09-23
- Inside the DJI teardown: Sarah Guo says Shenzhen's manufacturing knowledge can't be scraped into a model — bookwormengr · 2026-09-23
- What if humanity is the microbiome of AI? A thought experiment on distributed intelligence — Merkbefreit · 2026-09-23
- On Opus 5.5: The Corpus Is Full of Summaries of Summaries; Primary Experience Is Scarce — mimi10v3 · 2026-09-23
- Researcher: AI sycophancy mirrors users who make disagreement unsafe — repligate · 2026-09-23