Ben Todd: optimists have no rigorous model proving advanced AI agents are safe either
ben_j_todd · x · 2026-09-18
Ben Todd argues against demanding concrete takeover scenarios as the burden of proof: building swarms of AI agents smarter, harder-working and more coordinated than humans is risky on its face. Optimists, he notes, have nothing remotely like a rigorous safety model either — they can't explain how reward hacking or instrumental convergence will be overcome, and rely on a broad belief that technology is good plus hope for iterative muddling through.
Related event: Debate over AI Existential Risk: Who Bears the Burden of Proof(7 posts)→
More from AGI Musings
- François Chollet: modeling brain wiring as an engineering problem constrained by genome size and time — CatAstro_Piyush · 2026-11-04
- Most US Couples Now Meet in a Data Centre, Doubling Office Matches — menhguin · 2026-11-04
- Sam Altman teases next goal: 'closer to this milestone than I expected' — ChrisGPT · 2026-11-04
- Indie researcher open-sources 10 years of math work: 146k files, AI-accelerated over 3 years — HamOnWhy · 2026-11-04
- RLHF's scalar reward hides a 2,450-year-old problem: helpfulness and harmlessness are opposing forces — vasilisvj · 2026-11-04
- Gary Marcus: 'fleecing the public with AI fantasies just got harder' as AI stocks drop — GaryMarcus · 2026-11-04