Scale AI paper: 41 agent failure modes, new taxonomy to localize root causes
rohanpaul_ai · x · 2026-08-15
Scale AI released a paper 'Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures' proposing an interaction-centric taxonomy for agent debugging. It argues that when an AI agent fails, you should first locate where the first unrecovered failure happened: model, context, memory, tool layer, another agent, grader, or environment. The same visible mistake can have different causes; e.g., an ignored instruction might be due to the model failing to follow it or the harness dropping it during context compaction. The paper organizes 41 failure modes and tests on 40 examples. GPT-5.5 matched human labels with 80% accuracy and Cohen's κ 0.76; with 4 judges agreeing, precision reaches 96% but coverage drops to 68%. Practical advice: trace failure to the first broken component, then decide whether to retrain, fix harness, or repair environment.
More from Research
- Google-led paper uses sequential auctions to organize decentralized LLM agent workflows — _onionesque · 2026-10-16
- How meloform works: WASM pitch detection plus differentiable circuits learning 1,022 sine coefficients — henloitsjoyce · 2026-10-16
- Since OpenAI's math release, GitHub contributors keep driving down the integer-multiplication kappa — xennygrimmato_ · 2026-10-16
- RFdiffusion3 debuts in Science: generative AI designs proteins and nucleic acids atom by atom — AllThingsApx · 2026-10-16
- New Science report urges NSF to run experiments on science funding reform — alexeyguzey · 2026-10-16
- AI Turns Scrap Impurities into Useful Ingredients for Stronger Recycled Iron — webthing01 · 2026-10-16