Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training
Kunlun Zhu, Xuyan Ye, Yibo Li, Cheng Qian, Beibin Li, Heng Ji
cs.AI, cs.CL
2026-10-01
50,228 agent failure diagnoses with replay-tested corrections. Swapping the blamed action lifts replay pass from 18.4% to 51.1%; diagnosis SFT lifts Qwen3-8B exact-step agreement to 63.6%.
Failed agent rollouts today mostly get a negative outcome reward and are then discarded. A negative reward says the episode failed; it does not say which decision was wrong or what the agent should have done instead. Reusing failure requires two answers: where the error sits, and what action would have passed.
Existing failure-analysis resources each cover a slice. MAST classifies failure modes, Who&When attributes blame to agents, AgenTracer trains attribution models, but none ships an executed, verified correction, and none reruns the original action as a control. Diagnosis learning and action learning have never shared one pool of failures. AED fills that gap.
AED contains 50,228 error-diagnosis pairs from 9,961 source tasks, spanning 33 environments, 19 harness families and 23 policy models in text-based agents. Only natural failures are collected; no faults are injected. Environments range from BFCL, ALFWorld and τ-bench to SWE-bench Verified; harnesses from ReAct and OpenHands to AutoGen and LangGraph. The five-stage AET pipeline:
The replay control is the most valuable design choice here. Some tasks pass on retry alone: across 3,062 paired attempts, of the 1,564 first proposals that passed, the original-action retry also passed 470 times (30.1%). The net gain lives in the discordant cells, 1,094 correction-only passes against 93 retry-only. Without this control you cannot separate a useful correction from a branch that happened to pass.
| Comparison | Metric | Result |
| First-proposal correction vs original-action retry (3,062 pairs) | Verifier pass rate | 51.1% vs 18.4%, +32.7pp |
| Selected proposal after revision vs control | Verifier pass rate | 59.1% vs 13.7%, +45.3pp |
| Diagnosis-guided continuation vs generic reconsideration vs plain replay | Recovery success | 31.3% vs 26.0% vs 18.6% |
For diagnosis learning, full-diagnosis SFT on 1,656 source tasks lifts Qwen3-8B's exact-step agreement on a 943-case internal holdout from 47.2% to 63.6% over three seeds, above Claude Opus 5 at 54.7%, Gemini 3.8 Flash at 53.2% and GPT-6 Astra at 50.7%. Agreement rises monotonically across training sizes of 240, 480, 948 and 1,656 tasks, from 53.1% to 63.6%.
Actor training splits by environment. In a single-seed comparison, repair-containing recipes beat success-only SFT on WebShop-lite by 4.67 to 6.67 points, all three recipes lose roughly 8 to 10 points on TextQuest, and the preventive recipe loses to the untrained policy on real ALFWorld and ScienceWorld.
For anyone building agent training data, the value sits in the record structure more than raw scale: each record carries the trace, the diagnosis, an executed correction, the original-action control and an evidence grade, and slices cleanly into diagnosis, recovery and preference views. Matched replay sets a replicable quality bar for learning from failure, which pure trajectory dumps do not provide.
The cross-harness failure profiles carry a practical message: action errors lead in 8 of 9 harness families, while π agent concentrates on verification and observation (61.3% combined). Switching frameworks changes the error mix, so check error-type coverage when picking training data.
The policy gains are not yet portable. Treat this as a data asset to build on, not a recipe to copy.
Ranked by evidence strength: replay validation is the most solid, diagnosis learning next, actor training weakest.