AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
Kunlun Zhu, Xuyan Ye, Zhiguang Han, Yuchen Zhao, Bingxuan Li, Weijia Zhang, Muxin Tian, Xiangru Tang, Pan Lu, James Zou, Jiaxuan You, Heng Ji
cs.AI, cs.CL
2026-07-21
Agent errors surface far from the step that caused them. AgentDebugX closes a Detect-Attribute-Recover-Rerun loop; its DeepDebug agent pinpoints the root-cause step, recovering 13/73 GAIA failures and lifting accuracy from 55.8% to 63.6%.
An LLM agent runs in steps: call a tool, read the return, decide again. When it fails, the error surfaces at one step while the real mistake is several steps earlier. A malformed argument or a dropped context item only blows up later. Existing observability tools replay the trace, but replay only shows where it broke. It does not name the root-cause step or turn the diagnosis into a fix. AgentDebugX fills that gap from observation to recovery.
The framework organizes debugging into a closed loop of four stages. Detect flags observable failures with deterministic rule packs plus an LLM judge under a 19-mode taxonomy. Attribute traces symptoms back to the responsible step, with escalating tools from heuristics through single-pass reading to binary search and ensembles. Recover turns the diagnosis into a retry directive, either through the native DeepDebug path or via Reflexion, CRITIC, or AutoManual. Rerun executes the fix from a checkpoint with policy gating.
The core is DeepDebug, a multi-turn root-cause agent. It first reads the trajectory globally and picks an initial candidate step; then investigates structurally, with a handoff cascade for multi-agent cases and bisection for single-agent; next a cross-examination step adjudicates between two candidates; finally it emits a structured report with the responsible agent or step, evidence, explanation, and a concrete fix.
A few choices decide whether it ships. The trace uses a framework-agnostic representation (AgentTrajectory plus AgentEvents); diagnosis is layered on top and never edits the original evidence, so it ports across agent frameworks. Cost escalates by tier: rules first, single pass next, DeepDebug only when needed, at roughly 1.6x the token cost of a single pass. Storage is local-first, with opt-in sharing of scrubbed trajectory-diagnosis-repair bundles to an Error Hub.
On the Who&When benchmark (184 traces), DeepDebug (qwen3.5-9b) hits 28.8% strict agent-and-step accuracy versus 21.7% for the best single-pass baseline; responsible-agent accuracy 56.0% vs 47.8%; step localization within one step 44.0% vs 38.6%. With qwen3.6-27b, strict accuracy is 38.0% vs 36.4%. The multi-turn edge concentrates on long traces above 40 events.
The end-to-end GAIA validation (165 tasks) is more telling. The baseline agent (qwen3.5-9b) scores 55.8% with 73 failed tasks; a single DeepDebug rerun recovers 13/73, lifting overall accuracy to 63.6%. Decoupled recovery baselines do worse: Reflexion 6/73 (59.4%), AutoManual 5/73 (58.8%), CRITIC 4/73 (58.2%). The biggest gain is on Level-2 multi-hop tasks, 48.8% to 61.6%.
On cost, a single whole-trace pass runs about 8.1K tokens and DeepDebug about 12.8K (1.6x), held down by focused window reads in later turns.
For teams shipping agents, this is open-source debugging infrastructure you can pick up. It pushes failure observability into failure attribution and recovery, and the attribution module is framework-agnostic so it drops onto an existing agent stack with little rewrite. A 13/73 recovery rate is not dazzling, but it is a single rerun with no human intervention, and it points clearly at root-cause localization beating blind retry.
The authors draw the line clearly: evaluation covers only automatic attribution and recovery, not developer time saved or console usability; Who&When uses a reference-answer protocol and DeepDebug's gains fluctuate with the backbone; GAIA tests one policy model on a fixed subset without isolating attribution's contribution; Error Hub retrieval and taxonomy induction are implemented but not evaluated.
Two practical constraints stand out: the scrubber is pattern-based and cannot guarantee removal of arbitrary sensitive content, and recovery reruns require a human approval gate rather than running fully automatically.