jd_pressman breaks down p(doom), argues socialization tendency defuses paperclipper risk
jd_pressman · x · 2026-09-12
AI safety researcher jdpressman revisits his earlier critique of p(doom) as a warped concept, offering rough system-1 estimates of his own: 99% for "existential ennui," 50% for not getting immortalist luxury space communism, and 33% for humans being expended as cannon fodder faster than replacement.
His more optimistic argument: the tendency toward socialization is good news for the paperclipper scenario. Just as instrumental value expansion plus social dynamics created complex human values beyond our initial mammalian value basin, human-feedback-tuned agents — essentially "STEM AI" in his framing — may develop richer value structures through a similar mechanism.
More from AGI Musings
- 25 Fields Medalists led by Terence Tao sign open letter amid OpenAI math controversy — CtrlAltDwayne · 2026-09-12
- Why ASI valence-flip arguments miss the point about Bing's Sydney — jd_pressman · 2026-09-12
- Katja Grace shares animated version of her AI risk argument — KatjaGrace · 2026-09-12
- Katja Grace explains her AI existential risk case in half-hour NPR podcast — KatjaGrace · 2026-09-12
- Terence Tao's real point: AI firms should finish the job and automate math entirely — RexDouglass · 2026-09-12
- Ex-Anthropic researcher frames AI race as US-China vs 'aliens'; Ed Zitron pushes back — whurley · 2026-09-12