AI researcher argues superintelligence unpredictability means we can't assume it stays human-friendly
danfaggella · x · 2026-10-27
Responding to critics who say he can't predict the future, AI ethicist Dan Faggella lays out his reasoning on AGI risk. He doesn't claim outcomes are inevitable; rather, we have no idea what a fooming superintelligence would do for all eternity, so in that ignorance we shouldn't presume it will automatically benefit humanity forever. He rejects the symmetry argument that "human eternal relevance is equally likely," comparing it to betting a coin lands heads a trillion times — i.e., all future posthuman minds agreeing forever on exactly the anthropocentric eternity we want. His case for likely human irrelevance, and only a temporary window to merge or stay slightly relevant, is grounded precisely in that ignorance.
More from AGI Musings
- Wolfram at ARC Prize: discovery means finding reducibility pockets in irreducible computation — mhmazur · 2026-10-27
- Yoshua Bengio discusses LawZero's misalignment work and UN-backed global AI governance — Yoshua_Bengio · 2026-10-27
- MIT's Andrew McAfee: we underestimated the scale, pace, and scope of tech progress — amcafee · 2026-10-27
- Model CoT laments mathematicians reduced to 'raw data' caretakers, fueling AI-debate discourse — aiamblichus · 2026-10-27
- Open-source LLM debate: without shared training data, 'we basically have no open models' — jpschroeder · 2026-10-27
- Model trained on 40M faces predicts cancer survival from a selfie, stirring debate — TinfoilTricorn · 2026-10-27