Scale AI Paper Proposes Interaction-Centric Taxonomy to Localize Agent Failures
theomitsa · x · 2026-08-07
Scale AI released a paper, Model or Harness?, introducing an interaction-centric taxonomy for agent failures.
- The Problem: Existing agent evaluations often reduce failures to system-level outcomes, obscuring the exact origin of the fault and making targeted repairs difficult.
- The Method: The taxonomy localizes failures to broken edges between components such as models, tools, memory, context, graders, and environments.
- Actionable Routing: By assigning fault, it directs fixes appropriately—model-side failures point to post-training, harness-side failures to scaffolding/tool-integration, and grader/environment failures to evaluation redesign.
- Validation: The schema applies across agent architectures and was evaluated for reproducibility using independent reasoning agents as judges across frontier models.
More from coding & agent
- Opus 5.5 + Three.js demo stuns with water physics, fish and blossoms — nikola_mr64990 · 2026-10-16
- Grok-built linear regression bot claims $508K PnL in 125 days on Polymarket — Roger_M_Taylor · 2026-10-16
- Karpathy's knowledge engine workflow: Claude Code turns raw notes into a self-updating wiki — nikola_mr64990 · 2026-10-16
- Sierra and Meta unveil Personal Agent Protocol with Zapier as launch partner — sujingshen · 2026-10-16
- ArcKit adds 102 AI-narrated short films for commands across 14 community overlays — mcraddock · 2026-10-16
- Personal agents live or die on chaining tasks, not single-task ability — sujingshen · 2026-10-16