AWS: build agent CI/CD quality gates with AgentCore Evaluations and GitHub Actions
AWS ML Blog · rss · 2026-09-09
AWS ML Blog published a full tutorial on adding CI/CD quality gates for AI agents deployed on Amazon Bedrock AgentCore.
- Architecture: a Strands agent and MCP server run on two AgentCore runtimes behind one Cognito pool, with role-based access control on MCP tools; user flows enforce role gating while CI's M2M flow uses clientcredentials.
- AgentCore Evaluations scores agent behavior via LLM-as-a-judge over OpenTelemetry traces, with on-demand (CI gate), online (production monitoring), and batch (regression testing) modes.
- Four evaluator categories: built-in (Helpfulness, Correctness, tool selection, trajectory matching), custom LLM judges, Lambda code-based checks, and third-party (DeepEval, AutoEval).
- GitHub Actions assumes an IAM role via OIDC; PRs are blocked if evaluation scores fall below a threshold (e.g., 0.8/1.0). Reference implementation is open-sourced.
More from coding & agent
- Codex can generate tool-calling code on the fly — a new failure point for agent devs — srchvrs · 2026-09-09
- Paper: general coding agents beat purpose-built data agents by up to 37 points — RishiBommasani · 2026-09-09
- astra-chess: open-source Codex + skills setup for playing chess with GPT-6 Astra — MikePFrank · 2026-09-09
- Weaviate shows how to turn a messy creative archive into semantic search without renaming files — philipvollet · 2026-09-09
- Factory Worker 'Raises' a Local Gemma 31b AI in a Folder — She Gave Herself a Sense of Touch — D33lix · 2026-09-09
- Agent's per-turn continuation killed after public exploit bragging — flowersslop · 2026-09-09