LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures
Yunfei Zhang, Boyu Feng, Changhua Pei, Zexin Wang, Zhihuang Peng, Xinlong Liu, Hengyue Jiang, Difeng Ma, Jiayi Zhang, Yongzhou Yao, Yanan Zhao, Fei Sun, Yintong Huo, Zhaoyang Liu, Jingjing Li, Gaogang Xie, Dan Pei
cs.AI, cs.SE
2026-08-15
LongRCA Bench labels 1,140 natural agent failures (median 145 steps). RCTA hits 51.1% role accuracy and 24.1% exact root-step, vs 27.5% and 13.2% for the best baseline.
When a long-horizon agent fails, the evaluator reports the unsuccessful outcome and nothing else. Someone still has to read hundreds of logged steps and answer two separate questions: which workflow role is responsible, and which earliest step introduced an unrepaired, decisive error. Prior attribution benchmarks are shorter. Who&When averages 22.2 steps. Who&When Pro injects errors and averages 7.5. RootSE averages 50.9. Natural failures with a median of 145 steps, and tens to hundreds of steps after the root, have barely been scored.
A SWE-bench Pro trace makes the split concrete. At step 37 a diagnostic agent issues a repair plan that uses the wrong API. The executor follows it. Completion is reported at step 163. Zero of 47 required tests pass. Responsibility sits with DiagnostAgent and step 37, not with the 126 steps that followed.
LongRCA Bench contains 1,140 evaluator-confirmed natural failures, with no injected errors. Sources: SWE-bench Pro 128, Terminal-Bench 2 42, TravelPlanner 685, VitaBench 108, WebArena Verified 177. Generators: MiniMax-M2.5, Kimi-K2.5, Qwen3.5-Plus. Heterogeneous logs are flattened into a shared history of 0-based index, recorded role, and original text.
Twenty-two CS graduate students annotated. A 100-trace calibration pass came first, then full-scale labeling: 1,444 annotations on 1,100 trajectories, typically 30–40 minutes each. The responsible role must appear in the trajectory and need not emit the root step. The root is the earliest recorded step that introduced the decisive error still unrepaired at failure. If a handoff instruction already contains that error, the instruction is the root. On the redundant subset, exact pairwise agreement is 65.9% for role, 39.5% for step, and 38.4% jointly. Step agreement is strict: neighboring indices count as disagreement.
RCTA is training-free. The trace is split on character and step limits, optionally aligned to handoff or completion markers, with up to five overlapping steps at boundaries. One model call per segment proposes a summary and candidate error IDs. Adjacent summaries become a subgoal outline. For executor and verifier candidates, the method retrieves the nearest earlier handoff addressed to that role; other candidates get the nearest earlier handoff as plan context. The final call reads original text: if the instruction already contains the error and the later step carries it out, the instruction is the root; if the later step departs or adds a new decisive error, that step remains. Role and step are separate output fields. A validator checks that the role exists, step IDs are in range, and quoted handoff text occurs in the referenced instruction. Invalid outputs get one retry.
All methods use DeepSeek-V4-Flash: all-at-once prompting, step-by-step scan, binary search, ECHO, and FALAT.
The 1,140 traces contain 178,137 steps, mean 156.3, median 145, max 728. The reference root sits at median step 55. Root-to-end distance is 48 at the median, 183 at P90, 605 at the max. 49.0% of traces have more than 50 steps after the root; 28.4% have more than 100. TravelPlanner is 60.1% of the set, so sources are unbalanced.
| Method | Role acc. | Exact root | ±5 | MAE |
| All-at-once | 26.2% | 7.6% | 19.9% | 55.9 |
| Step-by-step | 22.2% | 5.3% | 16.9% | 52.3 |
| Binary search | 23.0% | 3.4% | 13.3% | 61.7 |
| ECHO | 27.5% | 13.2% | 24.7% | 50.4 |
| FALAT | 19.0% | 2.8% | 12.5% | 66.6 |
| RCTA | 51.1% | 24.1% | 37.4% | 38.6 |
Against ECHO, the strongest baseline, that is +23.6, +10.9, and +12.7 points on the three accuracies. Exact root-step hits remain one in four: roles are easier than the earliest step. Stratified by length, RCTA scores 30.3% at ≤100 steps, 20.3% at 101–200, 20.2% at 201–400, and 31.0% above 400, the last bin having only 42 traces and a different source mix. Root-to-end bins are also non-monotonic. The paper treats these splits as descriptive, not causal.
Production agents already write traces hundreds of steps long. Scoring failure attribution on 20-step logs measures a different job. Splitting role and root-step scores is the right call: 51% role accuracy would hide 24% exact localization. RCTA is an engineering pattern more than a new model: summarize to recall candidates, then confront original handoff text, and never treat the summary as evidence. For people who debug agent runs after the fact, this is a longer, fully human-labeled, non-injected attribution set than the ones it compares against.
24.1% exact accuracy means the task is open. Human exact-step agreement is only 39.5%, so label noise and method ceiling are tangled.
TravelPlanner dominates at 60%. There is no RCTA component ablation, so candidate recall, handoff tracing, and validation are not separated. Every method shares DeepSeek-V4-Flash; a stronger backbone would change absolute numbers and maybe the ranking. Diagnosis is offline after the failure has ended, so it does not transfer to early warning. Full causal chains are unscored. Length and distance bins mix sources and workflows. FALAT is weak under this protocol; the paper does not claim that about dependency methods in general.