Paper maps 41 agent failure modes to model, harness, tools and environment
omarsar0 · x · 2026-08-04
A new taxonomy maps 41 agent failure modes to the interaction where they arise
This paper argues that agent failures should not be reduced to a single “model bug” or “harness bug.” Instead, it classifies failures by the interaction edge where they originate—between model, harness, user, tools, memory, and environment—and assigns each failure a repair side.
What the taxonomy does
- Organizes 41 failure modes around interaction edges rather than components alone.
- Makes the framework actionable: model-side issues imply post-training fixes, harness-side issues point to scaffolding/tool-integration work, and environment or grader problems suggest evaluation redesign.
- Applies across coding assistants, long-horizon personal assistants, and multi-agent systems.
Evidence
- The authors ground the taxonomy in public benchmarks, model system cards, published reports, and logged trajectories.
- They also test whether the schema can be labeled automatically.
- Across four frontier models, the strongest judge reaches Cohen’s κ = 0.76 versus human labels, suggesting the taxonomy can support continuous production tracing rather than one-off postmortems.
More from coding & agent
- Context-window optimization is becoming essential for production AI agents — blaizedsouza · 2026-08-04
- Kimi K3 reportedly reaches 116 tok/s in a weird canvas-code eval — MaziyarPanahi · 2026-08-04
- openwiki v0.2.5 adds .openwikiignore to cut tokens and hide secrets from agents — BraceSproul · 2026-08-04
- After auditing 6 repos, a developer deleted 71% of their agent instructions — JuniorCustard4931 · 2026-08-04
- Build a Full AI Brand Asset Studio with Lovable in 5 Minutes — PrajwalTomar_ · 2026-08-04
- GitHub Stacked PRs repo shows how to split one big review into layered changes — DanWahlin · 2026-08-04