'A human approves everything' is a trap: approval fatigue and how to actually design safe agents
Efistoffeles · reddit · 2026-10-27
A long-form essay argues the real risk of AI automation is the human left in the loop. Drawing on Bainbridge's 1983 'Ironies of Automation,' it shows why 'a human approves everything so it's safe' fails: the better agents get, the less you check (approval fatigue), and errors compound — 95% per-step accuracy over 10 steps is only 60%. Better design: let agents do anything undoable without asking, gate only irreversible steps (send/delete/pay), and make each approval readable in five seconds. Case studies: open-source LetsFG gates a $2.5k booking behind a single purchase approval, while Coinbase's AgentKit gives agents their own wallets with no gating, caps, or allowlists — safety is yours to build.
More from AGI Musings
- Veteran engineer on what the AI-coding generation will never feel: 36-hour sprints and elusive bugs — bendee983 · 2026-10-27
- AI systems are not, and will never be, our friends, argues AI ethicist — LuizaJarovsky · 2026-10-27
- When AI agents pick your bank, brand feelings stop mattering, says VC — LexSokolin · 2026-10-27
- Reddit user argues OpenAI and Anthropic are holding back their best models — shadowt1tan · 2026-10-27
- OpenAI backs California SB 53 safety law as AI governance shifts into practice — Comfortable_Gene5180 · 2026-10-27
- Ben Goertzel: The AGI race is leaving much of AI research unexplored — bengoertzel · 2026-10-27