An End-to-End Agent Auditing Engine
Haoning Wang, Mingxun Zhang, Chenyue Yu, Yingjun Shang, Xia Hu, Guanchu Wang, Na Zou
cs.AI
2026-08-07
A2E runs one fixed model across 9 agent frameworks and 23 benchmarks. Frameworks differ little in correctness but vary widely in token cost, and none wins everywhere.
Harnesses, the frameworks that turn an LLM into a deployable agent (LangChain, CrewAI, OpenAI Agents SDK and the like), own the system prompts, tool interfaces, context management, and execution loop. Yet most agent evaluation looks straight through that layer at the underlying model, and the harness itself is rarely measured systematically.
Existing tools each cover only part of the job. The UK AISI's Inspect AI runs benchmarks but bolts harnesses on with per-framework adapters that can perturb native execution and erode trajectory fidelity. Arize's Phoenix offers observability but is not an end-to-end benchmark runner. Extending either to a new harness or benchmark means framework-specific engineering. A2E sets out to be a lightweight, minimally invasive, harness-agnostic substrate that unifies benchmark execution with faithful trajectory collection in one stack.
A2E is three layers.
The Task Layer sits on a homegrown Agent Task Protocol (ATP) that decouples benchmarks from harnesses. Each benchmark produces a TaskInput; each harness supplies an AgentRunner; ATP is the shared object interface between them. The payoff is that 23 benchmarks and 9 harnesses compose freely, with no glue code per pair.
The Monitor Layer builds on OpenInference and captures execution with OpenTelemetry spans (a bounded operation with start time, end time, status, and parent-child links). It instruments automatically, so no agent has to log itself, and it records the full reasoning to tool-call to observation to next-reasoning chain.
The Evaluation Layer introduces Lifecycle-Aligned Evaluation: every metric is registered under a stage of the agent lifecycle, namely Reasoning, Action, Final Answer, and Runtime Quality (efficiency and safety). Rule-based metrics (accuracy, success rate, latency, token usage, cost, step count) sit alongside LLM-as-judge metrics (reasoning quality, tool-use quality, instruction following, safety). Trajectories, metric definitions, and results all live in a database rather than scattered log files, so a new metric can be computed over already-stored trajectories without rerunning the agent.
The headline experiment runs all nine harnesses over all 23 benchmarks under one matched configuration: same backbone (DeepSeek-V4-Pro, FP4), same inference settings, tools, step limit, and timeout, five tasks per benchmark, 1,035 scored runs total.
| Dimension | Spread across 9 harnesses |
| Single-turn QA (arc, gsm8k, humaneval) correctness | Nearly identical; harness choice invisible |
| Multi-turn correctness | tau-bench 0.00-0.60, gdpval 0.00-0.60, traject-bench 0.20-1.00 |
| Token cost | Claude-Agent-SDK 2,063 to smolagents 7,319, a 3.5x spread, while correctness sits in 0.568-0.663 |
No harness dominates. openai-agents tops traject-bench at 1.00 but bottoms on tau-bench (0.20) and gdpval (0.00); llama-index leads the conversational benchmarks yet reaches only 0.40 on traject-bench. The overall leader, agno at 0.68, is propped up by the four sandbox benchmarks: on the 19 non-sandbox benchmarks alone, llama-index leads.
Swapping in GLM-5.2 for a cross-benchmark comparison (GDPVal, MMLU-Pro, tau3-bench), the success-rate gap reaches 0.20, 0.30, and 0.66, and the top-three harnesses are a different set on each benchmark.
The cleanest evidence is one tau3-bench task. LangGraph took 10,122 tokens, 4 model calls, and 3 tool calls, traced the fault to a suspended-overdue-bill account, and scored 1.0. CrewAI burned 96,704 tokens (about 9.6x), 9 model calls, and 5 tool calls resetting the APN, rebooting, and toggling airplane mode, never reached the account-level cause, and ended with zero signal and a 0.0.
For anyone choosing a stack, the direct signal is that agent performance is not just the model's job. Prompt construction, tool representation, context management, the execution loop, and termination policy can swing cost by multiples and flip success to failure. Pinning the model does not remove system-level variance. If cost matters, judging a harness on whether it got the right answer picks the wrong one; an evaluation that measures the whole trajectory, from planning through tools to efficiency and safety, is where the real differences show. Plainly, this is progress on evaluation infrastructure, not a new model or method, and the contribution sits at the engineering-measurement layer.
The authors concede that on this benchmark mix the rich process metrics "move little," and turning them into concrete guidance for harness design is left to future work. So the fine-grained metric framework demonstrates more value than it delivers as actionable design advice yet.
The sample is small: five tasks per cell, 0.20 resolution, high per-cell variance, and the authors explicitly state the table is not meant to rank the frameworks. Nine harnesses and 23 benchmarks do not cover the ecosystem. CrewAI's LLM spans carry no token counts, so it is omitted from the token-cost panel. The case study is a single task. Registry support does not imply every framework-benchmark pair passed end-to-end validation.