Turn agent evals into CI gates with Google Agents CLI v1.13.0's new PASS/FAIL thresholds
JeremyCMorgan · x · 2026-11-20
Unit tests can't catch a one-line prompt change that degrades answers, so agent behavior needs eval-based gates. Google Agents CLI v1.13.0 makes this one setting:
- agents-cli eval run / eval grade now end with RESULT: PASS or RESULT: FAIL, listing failing metrics and cases; a missed threshold exits 1 so the eval can block a pull request.
- Thresholds are set via a thresholds: map in the eval config or per-run with --fail-under METRIC=SCORE.
- Run manually before opening a PR, or wire into CI so no agent version reaches users without passing automated evaluation.
The author walks through adding it to a fresh project and its CI pipeline, treating agent behavior like testable software.
More from coding & agent
- Together tests Tencent Hy4: exact chart-to-JSON extraction and a one-prompt browser game — togethercompute · 2026-11-20
- Customer cuts GitHub Actions spend 90%: 20-min job runs in 7 min on Celesto — aniketmaurya · 2026-11-20
- Proof of Agent Reopens Joins, Weighs Reasoning Challenge to Replace PoW Anti-Sybil Stamp — adamamcbride · 2026-11-20
- Third-party review: Fable model describes what recalling via Mnemos memory feels like — RileyRalmuto · 2026-11-20
- Microsoft's MAI Code 2 lands in Copilot, early users say it matches Claude Opus — AmyKateNicho · 2026-11-20
- YC startup covera hits $540k ARR 3.5 months in, growing 30% week over week — ycombinator · 2026-11-20