Benchmark Exposes Fake Stats: 5 Codebase Tools Fail to Deliver 60% Token Savings
Obvious_Gap_5768 · reddit · 2026-08-06
The author conducted a rigorous benchmark of several codebase tools claiming 60–90% token savings, revealing that these marketing numbers collapse under real-world agent workloads.
Test Setup
- 15 Django questions from SWE-bench, 180 total runs across Claude Code and Codex.
- Same agent, prompt, and repository commit, with fresh indexes for every tool.
Key Findings
- Token Savings Exaggerated: Under Claude Code, the best tool saved only 16% output tokens, with some tools performing no better than baseline. Under Codex, the max saving was 35%.
- Harness Dictates Adoption: Claude Code loads MCP schemas on demand and often ignores tools, whereas Codex called every tool on every question. Tool adoption depends heavily on the harness rather than tool design.
- No Meaningful Quality Win: Quality differences across all setups were smaller than the evaluator's run-to-run noise.
- Deterministic Retrieval: In a deterministic retrieval benchmark without an LLM judge, tools showed varying accuracy in finding gold files, but indexing time costs were massive.
More from coding & agent
- Warp Launches Agent CLI: Multi-Model Coding Agent for Any Terminal — vikvang1 · 2026-08-06
- Developer Uses Codex Voice to Braindump and Auto-Assign Linear Tickets — OpenAIDevs · 2026-08-06
- OpenAI Showcases Codex Voice Mode: Pitch Ideas, Let AI Manage the Execution — OpenAIDevs · 2026-08-06
- Claude Builds 3D Downhill Racing Game with Zero External Assets via Procedural Generation — prasenx · 2026-08-06
- Data-Efficient Agent Distillation: Small Model Matches 9x Larger Teacher with Only 19 Samples — ctnzr · 2026-08-06
- YC-Backed Context.dev Launches API to Turn the Live Web into Structured Data for AI Agents — ycombinator · 2026-08-06