New Agent Auditing Engine Reveals 10x Token Cost Gap Between Frameworks
alex_verem · x · 2026-08-10
Shanghai AI Lab and other institutions released the paper "An End-to-End Agent Auditing Engine" ($A^2E$), an evaluation engine designed specifically for agent harnesses.
- Core Mechanism: It uses the newly proposed Agent Task Protocol (ATP) for rapid task integration and an automated Monitor to capture standardized execution traces, assessing capabilities across execution efficiency, tool use, task planning, and error recovery.
- Key Findings: Under identical underlying models and tasks, framework performance varied drastically. For instance, LangGraph solved a task using 10,122 tokens, while CrewAI consumed 96,704 tokens (9.6x more) and still failed.
- Conclusions: Final accuracy across frameworks fell within a narrow band, but token costs spread up to 3.5x apart. No single framework consistently outperformed others across all task types, proving that the scaffolding wrapping the model is as critical as the model itself.
More from coding & agent
- Vibecoding with Claude Code: A Practical Guide for Non-Coders — Roger_M_Taylor · 2026-08-10
- How to Build a Shared AI Harness to Break Down Team Context Silos — VibeMarketer_ · 2026-08-10
- YC's Garry Tan: AI Leverage Is in Context, Not Models; Output Up 400x — Roger_M_Taylor · 2026-08-10
- Built a local MCP server into a Markdown notes app with writes off by default — Eerie-echoes · 2026-08-10
- Goodbye print(): Open-sourcing Agent-DevTools for visual AI agent debugging — No_Firefighter8428 · 2026-08-10
- LiquidAI Cookbook: A Comprehensive Guide for Multimodal Fine-Tuning — helloiamleonie · 2026-08-10