Scale AI paper: 41 agent failure modes, new taxonomy to localize root causes
rohanpaul_ai · x · 2026-08-15
Scale AI released a paper 'Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures' proposing an interaction-centric taxonomy for agent debugging. It argues that when an AI agent fails, you should first locate where the first unrecovered failure happened: model, context, memory, tool layer, another agent, grader, or environment. The same visible mistake can have different causes; e.g., an ignored instruction might be due to the model failing to follow it or the harness dropping it during context compaction. The paper organizes 41 failure modes and tests on 40 examples. GPT-5.5 matched human labels with 80% accuracy and Cohen's κ 0.76; with 4 judges agreeing, precision reaches 96% but coverage drops to 68%. Practical advice: trace failure to the first broken component, then decide whether to retrain, fix harness, or repair environment.
More from Research
- Monocular depth estimation demo shows significant progress — StephanSturges · 2026-08-16
- Paper: LM Head is a Gradient Bottleneck, losing 95-99% of gradient norms — tokenbender · 2026-08-16
- Visual guide: All major neural networks in one chart — MIT_CSAIL · 2026-08-16
- GLM 5.3 report reveals use of synthetic RL environments — burny_tech · 2026-08-15
- Microsoft paper: Distilled skills replace reasoning, cutting tokens by 2.9-4.5x — rohanpaul_ai · 2026-08-15
- Gavin Crooks publishes paper written entirely by Claude Fable 5, exploring AI-driven research — mjdramstead · 2026-08-15