AgentBug-Smith paper: top coding agents fix just 9% of agent harness bugs
rohanpaul_ai · x · 2026-10-03
Researchers from UChicago, Fudan, Tsinghua and UIUC present AgentBug-Smith, an automated approach that discovers and reproduces real-world bugs in agent harnesses from open-source agentic systems, turning them into runnable tests and building a continuously growing Live-Harness-Bench (200 reproducible bugs so far).
Key findings:
- It beats general-software bug reproduction techniques by 10.67%–27.56% success rate across backbone LLMs.
- Harness bugs (tool calls, memory, prompts) depend on live model calls, making them hard to reproduce and test.
- The best of three coding agents fixed only 9% of these bugs, versus 40% on regular software bugs.
- A short guide distilled from past fixes lifted an agent from 1 to 6 correct fixes on 79 unseen bugs.
Practical takeaway: before trusting a coding agent with your agent's own code, test it on bugs you've already fixed.
More from coding & agent
- FT: US tech outages blamed on rush to fix bugs found by AI — nordicinst · 2026-11-19
- Claude Code overtakes Cursor and Copilot in job-post mentions ahead of Anthropic IPO — antgoldbloom · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS adds private CA support to Bedrock AgentCore Identity for connecting agents to self-hosted IdPs — AWS ML Blog · 2026-11-19
- Orchestrator's Cloud Agents graduate from beta to general availability — julianweisser · 2026-11-19
- Zero users, 33 pages shipped: dev crowdsources audit for AI-agent trust schema — TheHamzaShahbaz · 2026-11-19