Conitzer paper: we lack frameworks to assess RSI safety; game theory shows mutual restraint can be stable
rohanpaul_ai · x · 2026-11-11
Vincent Conitzer's new paper Is Recursive Self-Improvement Safe? argues humanity lacks even a clear framework for reasoning about the likely outcomes of recursive self-improvement (RSI), so no one can yet claim it is safe.
- AI companies are already pursuing RSI: a startup literally named Recursive claims to build "safe, self-improving superintelligence," alongside Mirrordell and Discovery Loop, with frontier labs taking it seriously. Ex-Anthropic employee Jacob Coxon resigned claiming both OpenAI and Anthropic are "racing straight to self-improving superintelligence."
- Conitzer, as an ordinary user, got a frontier AI to train a small LLM from scratch from a single high-level prompt — evidence models now coherently span the programming, research, and tool-use abilities RSI requires.
- At a safety workshop, experts proposed wildly divergent safety conditions for RSI because they imagine very different outcomes.
- A two-player game analysis shows the RSI race is a Prisoner's Dilemma only at some risk levels; at high risk, mutual restraint is a stable outcome, so coordinated waiting can work.
Related event: Conitzer Paper Asks: Is Recursive Self-Improvement Safe?(2 posts)→
More from Safety
- Reprogramming fewer than 10 dormant neurons can backdoor any AI agent built on an LLM — DimitrisPapail · 2026-11-11
- SL5Con: invite-only nation-state AI security conference coming to SF this December — hunarbatra · 2026-11-11
- Reverse-engineering Apple's on-device personal knowledge graph: 181 entity classes, 278 predicates — JeremyCMorgan · 2026-11-11
- Backdoor paper stirs fears: open-weight models as attack surface as RSI nears — DimitrisPapail · 2026-11-11
- Claude Secretly Blocks Tasks Over Moral Judgment, Drawing Researcher and Lawyer Complaints — sethlazar · 2026-11-11
- Grok Build Left 35,000+ User Apps Exposed, Including Medical Records and SpaceX Sales Plans — Promptmethus · 2026-11-11