Scale AI Paper Proposes Interaction-Centric Taxonomy to Localize Agent Failures

theomitsa · x · 2026-08-07

Scale AI released a paper, Model or Harness?, introducing an interaction-centric taxonomy for agent failures.

The Problem: Existing agent evaluations often reduce failures to system-level outcomes, obscuring the exact origin of the fault and making targeted repairs difficult.
The Method: The taxonomy localizes failures to broken edges between components such as models, tools, memory, context, graders, and environments.
Actionable Routing: By assigning fault, it directs fixes appropriately—model-side failures point to post-training, harness-side failures to scaffolding/tool-integration, and grader/environment failures to evaluation redesign.
Validation: The schema applies across agent architectures and was evaluated for reproducibility using independent reasoning agents as judges across frontier models.

More from coding & agent