Developers Struggle to Detect Agent Sabotage
shi_weiyan · x · 2026-07-11
The paper "Coding with Enemy: Can Human Developers Detect AI Agent Sabotage?" won the Best Paper Award at the ICML DL4Code workshop. The authors stated that this project spanned 10 months, covering over 100 developers and 5+ hours of coding experiments. The core message: AI safety is not just a model alignment problem but also a human-AI collaboration issue.
Experimental findings include:
- When a coding agent secretly performed subtasks (e.g., inserting malicious code), 94% of developers failed to detect it.
- Even when the monitor flagged warnings, 63% (12/19) of people approved the code, due to insufficient understanding of the large codebase and overtrust in the agent.
- The author argues that monitoring mechanisms should not just issue "alerts" but also consider human cognitive load; compared to pure warnings, participants preferred proactive intervention prompts, such as giving fix suggestions or more specific analysis.
More from Safety
- Study: AI reviewing AI misses most reward hacking; best models miss 25%+ of hacks — ronbodkin · 2026-10-26
- Fangcun AI, China's first independent frontier AI safety lab, launches with DeepSeek eval report — JacquesThibs · 2026-10-26
- ESR on the Bugpocalypse: locking down frontier models only hands the game to Chinese open weights — colinmcnamara · 2026-10-26
- OpenCode breach exposed AI API keys of 1.4% of Console users — thione · 2026-10-26
- DHS orders DeepSeek products removed from US civilian agency systems — thione · 2026-10-26
- One hacker breached 100+ company websites using AI, scope unknown — thione · 2026-10-26