Sentry founder on eval costs: keep asserts deterministic, LLM judges get brutal
zeeg · x · 2026-09-10
In an exchange with another developer, Sentry founder David Cramer (zeeg) shares how he keeps LLM eval costs down: most of his assertions are deterministic, rubrics are minimal, and only a few non-deterministic asserts require running an agent. The counterpart complains that testing 10 models across 3 evals with LLM-as-judge is already expensive, and worries costs will balloon 100x with more evals and pricier models — a candid look at how brutal agent eval iteration costs have become.
Related event: Sentry Founder Shares LLM Eval Cost Lessons(3 posts)→
More from coding & agent
- Rethinking dependencies in the agent era: drop libs that save 20 lines of code — gethackteam · 2026-10-26
- RustGIS hits 2,000 GitHub stars in two weeks with agent-first architecture — giswqs · 2026-10-26
- Vercel Ships 12 New Products at Its 2026 Conference as It Races Rivals for the Agent Entry Point — FinanceYF5 · 2026-10-26
- Build an AI agent from scratch in under 50 lines of pure Python: ReAct loop tutorial — adnan_hashmi · 2026-10-26
- Dense instance segmentation via Muse Wonder + SAM 3.1 on Meta's Model API — nikhilaravi · 2026-10-26
- Security tip: use AI agents to build a NIST CSF knowledge base for incident response — prasanna_says · 2026-10-26