A 1797 Goethe poem explains why AI researchers fear recursive self-improving systems
OmarUFlorez · x · 2026-09-16
In an op-ed published in El Comercio (Peru) and La Nación (Argentina), Omar U. Florez argues from a technical angle why so many AI researchers fear recursive self-improving models could escape control.
- He opens with Goethe's 1797 poem "The Sorcerer's Apprentice": the apprentice can command the broom to carry water but not to stop, and splitting it only multiplies the problem — the broom is precisely too obedient.
- Recent events back the concern: 27-year-old Anthropic researcher Jacob Coxon resigned, warning that Anthropic and OpenAI are advancing toward automatically self-improving systems; Anthropic alignment lead Evan Hubinger publicly backed him, saying he personally assigns over 10% probability to AI ending humanity within the next decade.
- Florez's core thesis: humans uniquely fear what they cannot control, and the alignment problem echoes the apprentice's two-century-old predicament.
Related event: Op-Ed: Why Researchers Fear Recursive Self-Improving AI(2 posts)→
More from AGI Musings
- Op-Ed: Why Researchers Fear Recursive Self-Improvement Could Run Out of Control — OmarUFlorez · 2026-09-16
- Mustafa Suleyman: model welfare is wrong, AI should not have rights or legal personhood — AlexTensor · 2026-09-16
- AI lets firms build compounding engines of institutional knowledge, says new essay — chetanp · 2026-09-16
- EA veteran: I held AI-risk beliefs 10 years before meeting anyone in the scene — AndyMasley · 2026-09-16
- Six experts on whether AI could wipe out humanity: less doom, still no consensus — gedbarker · 2026-09-16
- Analyst doubles down: humanoid robots may need major limits, possibly outright bans in many use cases — binarybits · 2026-09-16