Evals as a deployment gate: if a prompt change can't fail the build, you don't have evals
rseroter · x · 2026-10-09
Part 2 of a Stack Overflow Blog series argues most teams run LLM evals the way pre-CI teams ran tests: manually, in notebooks, after things break. Since a harmless prompt tweak or temperature change can silently tank quality on 8% of inputs, evals should work like tests—a CI gate that fails the build on regression. The piece lays out treating golden cases as version-controlled data (past failures never deleted), and pairs the deploy-time gate with drift detection to catch slow leaks in the weeks after shipping.
More from coding & agent
- Taming Codex's nagging auto-review with custom policies and agents.md guidance — pvncher · 2026-10-09
- LukeW: designers, we need a gorgeous mobile app for managing agents across devices — LukeW · 2026-10-09
- A Playable 3D Boat Game Embedded in an X Post, Built by Claude Opus — prasenx · 2026-10-09
- ProximalHQ's training setup lifts Qwen3.8-27B to 37.2% Pass@1 on DeepSWE, up 8.4 points — aryaman2020 · 2026-10-09
- edith-1 monitors agent traces, beats Sonnet-5.5 on balanced accuracy at 1/274 the cost — xennygrimmato_ · 2026-10-09
- How edith-1 works: probabilistic filter plus expensive agent judge for flagged runs — xennygrimmato_ · 2026-10-09