Sentry founder on eval costs: keep asserts deterministic, LLM judges get brutal
zeeg · x · 2026-09-10
In an exchange with another developer, Sentry founder David Cramer (zeeg) shares how he keeps LLM eval costs down: most of his assertions are deterministic, rubrics are minimal, and only a few non-deterministic asserts require running an agent. The counterpart complains that testing 10 models across 3 evals with LLM-as-judge is already expensive, and worries costs will balloon 100x with more evals and pricier models — a candid look at how brutal agent eval iteration costs have become.
Related event: Sentry Founder Shares LLM Eval Cost Lessons(3 posts)→
More from coding & agent
- Ben Bajarin: Frontier AI is becoming the enterprise cyber-defense stack — BenBajarin · 2026-10-26
- Building AI bots on raw APIs? Devs say models are 'dumb without the harness' — madeofAjala · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Microsoft demos intent-classifying agents built with Copilot App and Foundry, deployable across Teams and Outlook — marlene_zw · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26
- The identity crisis in software engineering: multiagent systems demand a new problem set — rakyll · 2026-10-26