Intelligence Has a Speed Limit: Control Theory Caps Recursive Self-Improvement
ArtificialOther · x · 2026-09-13
Google researcher Peyman Milanfar argues recursive self-improvement (RSI) has a hard speed cap rooted in control theory.
- Small-gain theorem: a feedback loop stays stable only if the product of its gains stays below one; RSI's appeal is precisely high gain — big modifications per unit of evidence.
- Historical precedent: 1980s–90s adaptive control had global stability proofs until MIT's Rohrs counterexample showed mildly violating assumptions causes systems to go haywire — often bursting unpredictably after running beautifully for a while.
- Self-improving AI has the same structure: its self-model is worst where it has least real-world experience, and aggressive optimization pushes deeper into those gaps.
- The bottleneck: the rate you can generate trustworthy evidence that a change was an improvement sets the maximum safe speed — intelligence has a speed limit.
A substantive technical rebuttal to overnight-superintelligence narratives.
Related event: Google Researcher Argues Intelligence Has a Speed Limit(5 posts)→
More from AGI Musings
- StudentBench: AI tutoring matches human GRE gains at 918x lower cost — gharik · 2026-10-31
- Box CEO Aaron Levie: AI progress is 'going vertical,' widening the frontier gap — rohanpaul_ai · 2026-10-31
- Model Collapse May Not Hit LLMs, But Perhaps Human Culture or Souls — generativist · 2026-10-31
- Nina Schick: who builds intelligence and how it's distributed is the defining question of our lives — NinaDSchick · 2026-10-31
- Martin Casado: AI's dazzling 80% makes everyone skip the hard 20% — martin_casado · 2026-10-31
- Cheap intelligence is like cheap bread — an ingredient list satire of mass-produced AI — jsuarez · 2026-10-31