A 1983 paper predicted the coding agent problem: users block only 5% of dangerous commands late in a session
Hesamation · x · 2026-11-07
Hesamation connects Lisanne Bainbridge's 1983 classic "Ironies of Automation" to today's coding agents.
- The automation playbook — automate everything — leaves humans with the two jobs they're worst at: watching for anomalies and taking over when the system fails.
- The paper notes nobody stays alert past 30-60 minutes on something that rarely breaks, and takeover skills atrophy.
Anthropic's own blog empirically confirms the alertness problem: when a dangerous command was hidden among Claude Code permission requests, users blocked only 17% early in a session and just 5% after 50+ prompts.
The paper's fixes: rely on alarms rather than operator attention, and keep skills alive with short manual-operation periods or simulators. Its kicker: the more advanced the control system, the more crucial the human operator's contribution.
More from AGI Musings
- Engineer warns: sacrificing the present entirely for the future is a dangerous game — KevinNaughtonJr · 2026-11-07
- Amazon reportedly trying to rehire employees laid off in AI-driven cuts — DavidLinthicum · 2026-11-07
- Peter Norvig on what's still worth learning when AI takes over the work — Diyi_Yang · 2026-11-07
- RareDiseaseBench debuts: 35 reconstructed rare disease cases test AI diagnostic agents — kenbwork · 2026-11-07
- Signal president debates AI policy head: should we stop trusting AI agents with our data? — neil_chilson · 2026-11-07
- Survey claims 68% of Americans support pausing AI and banning superintelligence — notkilleveryoneist · 2026-11-07