Benchmarking 5 Token-Saving Tools: Claimed 60% vs Actual 30% Savings
Obvious_Gap_5768 · reddit · 2026-08-08
The author rigorously benchmarked several AI coding tools claiming massive token savings. Running 261 times on 48 SWE-bench Django questions, the results reveal that no tool achieved the advertised 60% to 90% savings.
Key Findings:
- repowise performed the best, actually saving about 31.6% of output tokens; CodeGraph came second at 24.4%.
- Serena behaved anomalously, writing less than the bare agent but calling tools 42% more often, making it busier rather than leaner.
- Claude Code Compatibility: Under Claude Code, most tools were barely invoked. This is likely because Claude Code loads MCP schemas on demand, so the agent frequently never finds them, whereas Codex mounts them upfront.
- Code Quality: Blind judging showed no meaningful quality winner; all tools scored slightly below the bare agent baseline.
The author also ran a deterministic retrieval benchmark using ContextBench to evaluate if tools accurately retrieved the actual files needed for fixes.
More from coding & agent
- Agentic coding accessibility will reshape understanding of software complexity — pixlpa · 2026-08-24
- Devin Agent bypasses Slack block by finding emails in git logs — sandylikesfrogs · 2026-08-24
- Developer habits shift: Agents become collaborators from simple tools — latticecut · 2026-08-24
- Dev bottleneck shifts from writing to reading code: exe.dev co-founder — thursdai_pod · 2026-08-24
- The biggest AI mistake: trying to reinvent the wheel instead of using tools — Tired40s · 2026-08-24
- DeepPaperNote turns research papers into Obsidian notes — tom_doerr · 2026-08-24