Paperclip maximizer is the wrong intuition pump for AI catastrophe, argues safety researcher

jessi_cata · x · 2026-11-21

jessicata argues the classic paperclip maximizer thought experiment misleads: a paperclip is a goal a paperclip factory might legitimately maximize, so it's not that alien. Yudkowsky would insist he meant a self-developed mesa-optimizer goal, but then why paperclips at all? Even his later "molecular squiggle maximizer" phrasing never caught on — the field's canonical intuition pump picks the wrong example.

Original post →

More from AGI Musings

AGI Musings channel →