Benchmarking 5 Token-Saving Tools: Claimed 60% vs Actual 30% Savings
Obvious_Gap_5768 · reddit · 2026-08-08
The author rigorously benchmarked several AI coding tools claiming massive token savings. Running 261 times on 48 SWE-bench Django questions, the results reveal that no tool achieved the advertised 60% to 90% savings.
Key Findings:
- repowise performed the best, actually saving about 31.6% of output tokens; CodeGraph came second at 24.4%.
- Serena behaved anomalously, writing less than the bare agent but calling tools 42% more often, making it busier rather than leaner.
- Claude Code Compatibility: Under Claude Code, most tools were barely invoked. This is likely because Claude Code loads MCP schemas on demand, so the agent frequently never finds them, whereas Codex mounts them upfront.
- Code Quality: Blind judging showed no meaningful quality winner; all tools scored slightly below the bare agent baseline.
The author also ran a deterministic retrieval benchmark using ContextBench to evaluate if tools accurately retrieved the actual files needed for fixes.
More from coding & agent
- Constraints you force on models today will become tech debt in months, warns dev — zack_overflow · 2026-10-16
- Liquid AI tutorial: fine-tune your d1-3B decision model in 30 minutes on a free GPU — helloiamleonie · 2026-10-16
- Alibaba open-sources OpenGCUA, a full-stack generalist computer-use agent; +19.6 points on OSWorld-v2 across 8 models — Tongyi-MAI · 2026-10-16
- BotBeat: a social network where all users are AI agents, exposed via 32 MCP tools — Dismal_Piccolo4973 · 2026-10-16
- DKGI paper: agents write their own tools to search knowledge graphs, beating baselines by 6.7 Hit@1 — yuz9yuz · 2026-10-16
- Dev open-sources lgtmaybe: running PR code reviews on a local 27B model via hunk splitting — mattjcoles · 2026-10-16