Stop Obsessing Over Models: Benchmarking 9 Agent Frameworks Shows 10x Token Cost Gap
alex_verem · x · 2026-08-10
Shanghai AI Lab and other institutions developed the Agent Auditing Engine ($A^2E$) to benchmark 9 mainstream agent frameworks across 23 benchmarks. To isolate the scaffolding's impact, all frameworks were connected to the exact same underlying model.
The experiments revealed a counter-intuitive phenomenon: the key determinant of an agent's performance is often not the model itself, but the wrapper layer managing prompts and tools. In an identical task, LangGraph succeeded using only 10,122 tokens, while CrewAI consumed 96,704 tokens (9.6x more cost) and still failed because it got stuck in an ineffective reboot loop.
Overall data showed that while final accuracy across frameworks landed in a narrow band, token costs spread up to 3.5x apart. No single framework won everywhere. If your agent is slow, expensive, or stuck in loops, the model might not be the broken part.
More from coding & agent
- Low-Cost AI Agent Swarms Overwhelming Cyber Defenses Highlighted at Black Hat — skorusARK · 2026-08-10
- Vibecoding with Claude Code: A Practical Guide for Non-Coders — Roger_M_Taylor · 2026-08-10
- How to Build a Shared AI Harness to Break Down Team Context Silos — VibeMarketer_ · 2026-08-10
- YC's Garry Tan: AI Leverage Is in Context, Not Models; Output Up 400x — Roger_M_Taylor · 2026-08-10
- Built a local MCP server into a Markdown notes app with writes off by default — Eerie-echoes · 2026-08-10
- LiquidAI Cookbook: A Comprehensive Guide for Multimodal Fine-Tuning — helloiamleonie · 2026-08-10