Andrew Critch: stop calling extinction risks "improvement"—RSM vs RSI matters
AndrewCritchPhD · x · 2026-11-12
AI safety researcher Andrew Critch argues against industry framing that brands extinction risks as "improvement," and offers a sharper conceptual vocabulary:
- Recursive Self-Modification (RSM): AIs modifying their next versions, including their self-modification tendencies—an AI that kills all humans is still engaged in RSM;
- Recursive Self-Improvement (RSI): RSM whose modifications follow a trend of "improvement," which is value-dependent; extinctive RSM is not improvement for most humans;
- Runaway RSM: self-modification escaping human control—uncontroversially extremely dangerous;
- RSI attempts can fail and yield runaway RSM instead, so extinctive AI development shouldn't be called "improvement."
He provides concrete rebuttal phrasings for both AI-skeptical and RSI-sympathetic stances, urging people not to play along with the branding.
More from AGI Musings
- Kai-Fu Lee wraps 12-hour London blitz promoting 'AI Native' book launch — kaifulee · 2026-11-12
- Do closed AI labs' research agendas diverge? Talent flow acts as a penalty term — stochasticchasm · 2026-11-12
- AI 'doomers' call themselves 'the Cassandra fringe' — critic says the analogy flatters overconfidence — DrTechlash · 2026-11-12
- AI agent traffic reportedly grew 8x in just two months — Dr_Singularity · 2026-11-12
- Ten gems from Diogo's Code Summit talk: RLHF is part of the problem, software is back — TheTuringPost · 2026-11-12
- AI models can prove anything, making 'mathematically trivial' ICLR reviews obsolete — nanjiang_cs · 2026-11-12