Sentry Founder Shares LLM Eval Cost Lessons

Sentry co-founder David Cramer shared his LLM eval engineering practices, favoring deterministic assertions over costly LLM judges—a recent metaloop run burned $200 in an hour. He keeps rubrics minimal, reserving LLM-based checks for a few non-deterministic cases.

2026-09-10 ~ 2026-09-10 · 3 related posts

• Sentry founder: one metaloop eval run burned $200 in an hour — zeeg · 2026-09-10
• Sentry founder on eval costs: keep asserts deterministic, LLM judges get brutal — zeeg · 2026-09-10
• Sentry CTO: most of my CI assertions are deterministic, LLM rubrics kept minimal — zeeg · 2026-09-10