Failing LLM Evals Cause Production Regression
larabyeol · reddit · 2026-07-11
The author shares a real-world case: a team started LLM automated evaluations in November, using GPT-4o as a judge to score deployments across 8 dimensions.
Initially, it caught obvious regressions well. But later, a system prompt tweak broke an edge case, and the judge still scored it 8.7/10, impacting about 3% of users post-launch. The issue wasn't found until support tickets came in the next day.
In retrospect, the author identifies two main causes:
- The judge prompt and business prompt lacked unified version control, causing the judge itself to drift.
- Even with better versioning, automated judges struggle to catch subtle quality regressions, which might be a methodological limitation rather than just a tooling issue.
They later moved both into the same versioned system, at least making drift visible before deployment.
More from coding & agent
- Coding agents are heading toward an AI-writes, AI-reviews, human-approves workflow — aftahi_ai · 2026-07-22
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22
- GitHub review bot hits its PR limit and forces a 39-minute cooldown — DanielLockyer · 2026-07-22
- Max reasoning effort appears to be mobile-only in Codex Remote, not desktop — GabGarrett · 2026-07-22
- A Reddit demo argues online stores should expose carts and pricing through MCP — gelembjuk · 2026-07-22
- Open-source AI SDK provider routes Vercel apps through a local Codex subscription — lgrammel · 2026-07-22