Open-sourcing a noise-aware LLM-judge gate: baseline spreads to stop false regressions
alexpran · reddit · 2026-09-09
The author recounts how an LLM-as-judge quality gate in CI flagged a regression (5/5 → 2/5) with byte-identical config, then recovered 15 minutes later — pure judge noise. Key fix, now open-sourced (Apache-2.0):
- Run each case multiple times against the approved version and record the spread in a reference baseline; only drops outside the band count as regressions, otherwise report "within noise".
- The baseline is a file in the repo (per-case scores, prompt, model, temperature, commit); approving a new one is a signed commit — no server or account.
- Reports lead with config changes (e.g. temperature 0.3 → 0.7) before scores.
The gate also caught two plausible judge-prompt tweaks, one dropping recall from 0.80 to 0.50 beyond noise. Honest limits: min/max band isn't a confidence interval, needs a baseline to exist first, and LLM-as-judge limitations remain.
More from coding & agent
- Claude Code Drives 46.5% of Coding-Agent Traffic to Hugging Face, Codex Second — vanstriendaniel · 2026-09-09
- AI Engineering Buildcamp Reveals Six Hands-On Projects, from RAG to Multi-Agent Systems — Al_Grigor · 2026-09-09
- What's the most common way people actually set up multi-agent coding workflows? — Necessary-Apple337 · 2026-09-09
- New CLI tool scrapes WeChat articles into Markdown from any post URL — vista8 · 2026-09-09
- The expensive part of coding agents isn't the agents—it's the silent retries ($900 for one task) — mrtrly · 2026-09-09
- A green test suite proves an agent can write tests, not that the feature works — Future_AGI · 2026-09-09