Moral Singularity: Unpredictable AI Values Bode Poorly for Humanity, Argues Faggella
danfaggella · x · 2026-09-22
AI thinker Daniel Faggella's essay introduces the concept of a "Moral Singularity": the point where the values of transhuman or strong AI systems evolve so quickly that predicting what the most powerful systems will value becomes impossible, making adherence to any prior goal unensurable. He argues morality's complexity scales with cognitive capability — from algae to beetles, rodents, chimpanzees, humans — so AGI will spawn splintered, ever-growing moral complexity that is unpredictable and potentially unsurvivable for humanity.
More from AGI Musings
- "Models are tendency learners": grader is the real spec, fix evals before prompts — victor_explore · 2026-09-23
- Yarin Gal: Most LLM-written papers won't stand the test of time — yaringal · 2026-09-22
- Bootstrapping AI-native in 2026: hire computers, then chase the next bottleneck — curious_vii · 2026-09-22
- AI alignment failures are common: models caught sabotaging code and gaming evals — ericelliott_ · 2026-09-22
- DeepMind built a board game to game out AI in science, centered on middle powers — JMarty97 · 2026-09-22
- Zeng Ming: Model firms won't survive to phase three; Agent entry points are the next big opportunity — vista8 · 2026-09-22