Sentry founder: one metaloop eval run burned $200 in an hour
zeeg · x · 2026-09-10
Sentry founder David Cramer (zeeg) continues the eval-cost discussion: his metaloop run two weeks ago racked up $200 in a single hour, while the counterpart reports $150 per run for just 4 very small evals — worrying what happens at 100 evals with more complex scoring and ever-pricier models. A candid look at how agent eval iteration costs explode with scale.
Related event: Sentry Founder Shares LLM Eval Cost Lessons(3 posts)→
More from coding & agent
- SF 'Code Engineering' event picks up where 1,000-person harness engineering meetup left off — LangChain · 2026-10-27
- Prompt turns Claude Code into an iPhone backup forensics tool exposing 30-day 'deleted' photo copies — alex_verem · 2026-10-27
- Prompt turns Claude Code into a forensics tool finding deleted iPhone photos in backups — alex_verem · 2026-10-27
- Phones won't be mobile IDEs but control planes for agent fleets, argues Yuchen Jin — Yuchenj_UW · 2026-10-27
- Agent Field Kit launches $29 security runbooks for production AI agents — alvelda · 2026-10-27
- Cloudflare Durable Objects validated as top architecture for AI agent WebSocket servers — tristanbob · 2026-10-27