Open-source benchmark tracks coding-agent token usage without leaking prompts
bestofdesp · reddit · 2026-07-28
I open-sourced agent-token-bench, a free MIT-licensed CLI for analyzing local Codex rollout files and measuring coding-agent token usage without exposing prompts, responses, reasoning, tool arguments/results, or workspace paths.
What it measures
- Cached vs. uncached input
- Output and reasoning-output totals
- Credit-equivalent usage via a dated rate card
- Tool calls, polling, and compactions
- Per-turn and per-non-poll-action efficiency
- Before/after experiment deltas
Notes
- The repo includes an anonymized initial observation, but it is explicitly framed as directional rather than a formal A/B result.
- The author asks for feedback on methodology, Claude Code support, safe on/off intervention switching, testing Ponytail as a distinct arm, and what quality outcomes should be required before claiming savings.
More from coding & agent
- PIRL adds closed-loop verification to RL post-training and improves accuracy on reasoning and code tasks — This_Ad9834 · 2026-07-28
- MyClaw pitches 30-second setup and 24/7 hosting for AI agents — manishkhosiya · 2026-07-28
- Microsoft open-sources a governance toolkit for autonomous AI agents — microsoft · 2026-07-28
- A Python repo turns technical book PDFs into Claude Code skills — virgiliojr94 · 2026-07-28
- Hugging Face open-sources a local speech-to-speech voice-agent stack — huggingface · 2026-07-28
- New APIs let agents react to real-world events in real time — jarrodwatts · 2026-07-28