Why Fast Takeoff via RSI Is Unlikely: Human Approval Is the Bottleneck
GarrisonLovely · x · 2026-09-18
Garrison Lovely argues that a truly crazy "fast takeoff" requires taking humans entirely out of the loop—voluntarily handing control of successive AI generations to AIs themselves. Human approval would become the rate-limiting factor, since meaningfully understanding, redirecting, or stopping the loop takes time.
He notes US AI companies treat RSI (recursive self-improvement) as the most important milestone in the race—the point where a leader converts a temporary lead into a permanent one—while it has long been recognized as a uniquely dangerous threshold where an AI could secretly pass undesirable traits to its successors. The thread also questions whether any rival would realistically get to RSI first.
More from AGI Musings
- From Curing Cancer to Killing You: How the AI Bubble Narrative Keeps Flipping — GCWebDesigner · 2026-09-18
- After Automation: Why Preserving 'Productive Friction' Matters in the AI Age — every · 2026-09-18
- Anthropic reveals Claude now leads 26% of its own R&D, up from near zero 6 months ago — Outside-Iron-8242 · 2026-09-18
- Noam Brown: Air-gapping may not stop misaligned AI; safety monitoring eats 20% extra compute — aronchick · 2026-09-18
- 36 of 38 gene companies shipped 1918 flu genome, fueling AI bioweapon debate — BlackHC · 2026-09-18
- The Yudkowskian error: intelligence is explanation generation, not goal maximization — inductionheads · 2026-09-18