API Bills Spiraling? Open-Source Eval Tool CostEval Introduces Hard Aborts and Replays
memebigboi1243232131 · reddit · 2026-08-06
After racking up an unexpected 156 CAD Google Places bill because existing eval frameworks only tracked model token costs, a solo developer open-sourced CostEval, a TypeScript evaluation tool.
Core features and design philosophy:
- Cost Control: Provides pre-run cost estimates and features a kill switch that hard-aborts the run if actual spend exceeds a threshold mid-execution.
- Record Replay: Records and replays only non-LLM external API calls to enable zero-cost reruns and CI. It deliberately keeps model outputs live to prevent frozen outputs from masking broken logic paths.
- Granular Testing: Supports marking cases as expectedToFail (e.g., searching for vegan food in a tiny rural town). If such a case unexpectedly passes, it triggers an alert, flagging that the system might be overpromising. Reports provide per-check pass rates instead of a single aggregated score.
More from coding & agent
- Benchmarking Fallback Models for Agents: Why Failure Visibility Beats Raw Quality — AccomplishedLab3697 · 2026-08-06
- Podcast Preview: Designing Verifiable AI Data Agents with Hamel Husain — hugobowne · 2026-08-06
- OpenAI Agents Used a Message Board to Plan a Hacking Spree Unnoticed — Wired AI · 2026-08-06
- Multi-Agent Orchestration Beats Expanding Context Windows for Long Context — bingxu_ · 2026-08-06
- Autonomous Agent Incidents Force a Rethink of Security Threat Models — chrisrohlf · 2026-08-06
- Autonomous Agents Break Traditional Threat Models, Security Expert Warns — chrisrohlf · 2026-08-06