Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
Harsh Raj, Vipul Gupta, Anas Mahmoud, Razvan-Gabriel Dumitru, Darvin Yi, Aakash Sabharwal, Yunzhong He
cs.AI
2026-07-31
A 41-mode taxonomy from Scale AI localizes each agent failure to an edge between two components and a fault side, telling you whether to fix the model, harness, or environment; top judge reaches κ=0.76.
When an agent run fails, the log shows a system-level outcome: the task didn't finish, the wrong email went out, the code didn't run. But the same visible symptom can call for very different fixes: retraining the model, reworking the harness (tool integration, memory, context management), fixing a buggy evaluation environment, or repairing the benchmark itself. The authors call this the "repair-assignment problem": identical failures, different root causes, different fixes.
Most existing failure taxonomies are fine-grained lists built for a single benchmark and do not transfer across systems. This paper wants a shared structure that works across architectures and is directly actionable: once you classify a failure, it tells you which component to change.
The authors decompose an agent system into a focal model plus surrounding components, grouped into three families:
Each failure is localized to an edge between two components and tagged with a fault side, which end of the edge owns the repair. Model-side failures point to post-training targets; harness-side failures point to tool-integration or context-management fixes; environment- or grader-side failures mean the evaluation conditions themselves must be redesigned.
The attribution rule matters: trace backward from the system-level failure to the "earliest failure from which execution does not recover" and treat that as the root cause; later errors count as consequences. Of the 41 failure modes, 36 are model-attributable and 5 land on surrounding components.
The taxonomy is grounded in 40 worked examples (SWE-bench, ClawsBench, Harbor-Mix, Anthropic system cards, GitHub issues, agent trajectories). A representative case: an agent deleted more than 200 emails, but the root cause was not the model "wanting" to delete them. Context compaction dropped the constraint telling it to leave the inbox alone, a textbook context-side harness fault.
Reproducibility is tested with an agent-as-judge setup: each judge receives a reference to the failure source (via Docent or Face), reconstructs the evidence itself, then classifies. Four frontier models serve as independent judges:
| Judge | κ vs human category labels |
| GPT-5.5 | 0.76 |
| Claude Opus 4.6 | 0.71 |
| Claude Opus 4.7 | 0.71 |
| Claude Opus 4.8 | 0.70 |
Pairwise agreement peaks at κ = 0.84 (Opus 4.6 vs 4.8). Requiring all four judges to agree lifts category precision to 0.96, but coverage drops to 68%.
For anyone building agents, this answers "where to look after a failure," not just "did it fail." The fault-side split maps straight onto which tool or team should intervene, and it transfers across architectures: coding assistants (Claude Code, Codex), long-running personal assistants that read mail and browse, and multi-agent systems. A judge agreement of κ = 0.76 suggests the categories capture genuine shared structure rather than one annotator's preferences.
Thirty-six of the 41 modes land on the model, a heavy model-side skew the authors acknowledge; this risks underweighting harness and environment failures, which in practice are where many agent problems actually live. The 40 worked examples are curated, not a random sample of failures, so there is selection bias toward clear, interesting cases. A κ of 0.76 is "substantial" agreement, far from perfect, and category-level agreement is better than fine-grained failure-mode-level. The taxonomy is a localizer, not a fixer: it names the edge, not the specific patch. Finally, the "earliest unrecovered failure" rule gets ambiguous in long trajectories where many errors interleave.