Intelligence Has a Speed Limit: control theory caps recursive self-improvement
docmilanfar · x · 2026-09-13
Google research scientist Peyman Milanfar's essay applies control theory to recursive self-improvement (RSI) in AI:
- RSI is often framed as almost magical — a model improves itself iteratively toward superintelligence. Control theorists see a specific flaw in that expectation.
- Any self-improving loop has a maximum safe speed, set by how fast you can generate trustworthy evidence that a change was actually an improvement.
- The small-gain theorem says a feedback loop stays stable when the product of its gains stays below one — verification speed, not raw capability, is the binding constraint.
Implication: self-improvement isn't impossible, it's speed-limited by how quickly changes can be validated — a hard constraint often missing from AGI acceleration arguments.
More from AGI Musings
- Math frontier will move with AI, but dirty proofs are unacceptable — njyx · 2026-09-13
- Dario Amodei's new essay urges pacing AI frontier, pledges permanent third-party access — dhadfieldmenell · 2026-09-13
- UK parliament hears warnings AI could kill all humans within a decade, with >10% risk cited — connoraxiotes · 2026-09-13
- Skeptic picks apart the 'AI copies itself' doomsday scenario: where are the details? — recallingmemories · 2026-09-13
- David Deutsch, Nick Bostrom and more debate whether LLMs can reach AGI at Oxford — anderssandberg · 2026-09-13
- Regulation is coming for open and closed AI models alike — the question is proactive or reactive — benjamin_warner · 2026-09-13