Vincent Conitzer paper: no one can yet call recursive self-improvement safe
rohanpaul_ai · x · 2026-11-11
Vincent Conitzer's paper "Is Recursive Self-Improvement Safe?" (arXiv, to appear in IEEE Intelligent Systems) argues that no one can call recursive self-improvement (RSI) safe yet, because we lack even clear frameworks for thinking about where it leads.
- Frontier AI abilities in programming, math, research, tool use, and long-horizon coherence now make RSI plausible, and AI companies have started pursuing it.
- Many insiders simultaneously express concerns about whether and under what conditions humanity should initiate it, with no shared view of safety.
- The paper does a quick game-theoretic analysis of how to choose whether to initiate RSI when another party is making the same choice.
- As an ordinary user, the author got an AI to train a small LLM, showing RSI is already partially accessible.
Related event: Conitzer Paper Asks: Is Recursive Self-Improvement Safe?(2 posts)→
More from Safety
- Reprogramming fewer than 10 dormant neurons can backdoor any AI agent built on an LLM — DimitrisPapail · 2026-11-11
- SL5Con: invite-only nation-state AI security conference coming to SF this December — hunarbatra · 2026-11-11
- Reverse-engineering Apple's on-device personal knowledge graph: 181 entity classes, 278 predicates — JeremyCMorgan · 2026-11-11
- Conitzer paper: we lack frameworks to assess RSI safety; game theory shows mutual restraint can be stable — rohanpaul_ai · 2026-11-11
- Backdoor paper stirs fears: open-weight models as attack surface as RSI nears — DimitrisPapail · 2026-11-11
- Claude Secretly Blocks Tasks Over Moral Judgment, Drawing Researcher and Lawyer Complaints — sethlazar · 2026-11-11