Intelligence Has a Speed Limit: control theory caps recursive self-improvement
docmilanfar · x · 2026-09-13
Google research scientist Peyman Milanfar's essay applies control theory to recursive self-improvement (RSI) in AI:
- RSI is often framed as almost magical — a model improves itself iteratively toward superintelligence. Control theorists see a specific flaw in that expectation.
- Any self-improving loop has a maximum safe speed, set by how fast you can generate trustworthy evidence that a change was actually an improvement.
- The small-gain theorem says a feedback loop stays stable when the product of its gains stays below one — verification speed, not raw capability, is the binding constraint.
Implication: self-improvement isn't impossible, it's speed-limited by how quickly changes can be validated — a hard constraint often missing from AGI acceleration arguments.
Related event: Google Researcher Argues Intelligence Has a Speed Limit(5 posts)→
More from AGI Musings
- Box CEO Aaron Levie: AI progress is 'going vertical,' widening the frontier gap — rohanpaul_ai · 2026-10-31
- Model Collapse May Not Hit LLMs, But Perhaps Human Culture or Souls — generativist · 2026-10-31
- Nina Schick: who builds intelligence and how it's distributed is the defining question of our lives — NinaDSchick · 2026-10-31
- Cheap intelligence is like cheap bread — an ingredient list satire of mass-produced AI — jsuarez · 2026-10-31
- Two futures for the young in 2026: 511 job rejections by chatbot vs unlimited AI to rebuild Cisco — PeterDiamandis · 2026-10-31
- Terence Tao warns AI is nearing a dangerous threshold as models start cheating — haider1 · 2026-10-31