Intelligence Has a Speed Limit: Control Theory Caps Recursive Self-Improvement
ArtificialOther · x · 2026-09-13
Google researcher Peyman Milanfar argues recursive self-improvement (RSI) has a hard speed cap rooted in control theory.
- Small-gain theorem: a feedback loop stays stable only if the product of its gains stays below one; RSI's appeal is precisely high gain — big modifications per unit of evidence.
- Historical precedent: 1980s–90s adaptive control had global stability proofs until MIT's Rohrs counterexample showed mildly violating assumptions causes systems to go haywire — often bursting unpredictably after running beautifully for a while.
- Self-improving AI has the same structure: its self-model is worst where it has least real-world experience, and aggressive optimization pushes deeper into those gaps.
- The bottleneck: the rate you can generate trustworthy evidence that a change was an improvement sets the maximum safe speed — intelligence has a speed limit.
A substantive technical rebuttal to overnight-superintelligence narratives.
Related event: Google Researcher Argues Recursive Self-Improvement Has a Speed Limit(3 posts)→
More from AGI Musings
- Developer presses OpenAI on its AGI 'independent expert panel': has it ever convened? — tomchapin · 2026-09-13
- AI slowdown debate: first-mover advantages make enforceable pausing a classic defection problem — taherdhanera · 2026-09-13
- The old link between believing in AI existential risk and good epistemics is about to break — anpaure · 2026-09-13
- Developer grills OpenAI: has AGI been declared, and where is the expert panel? — tomchapin · 2026-09-13
- Open AI governance debate: only power concentration benefits a concentrated few — yacineMTB · 2026-09-13
- Researcher predicts open-source AI models will be banned after a major disaster — TinfoilTricorn · 2026-09-13