AWS pairs LLM-as-a-Judge quality scoring with an autonomous DevOps agent to monitor production multi-agent systems
AWS ML Blog · rss · 2026-09-12
An AWS engineering blog argues that production multi-agent failures often slip past traditional infrastructure monitoring: a poorly scoped supervisor prompt raises no errors but routes 20% of requests to the wrong specialist, and a missing IAM permission can make an agent return empty responses while logs show only successful tool calls. CloudWatch proves systems executed, not that agents helped users.
Using a four-specialist airline reservation system (multi-city bookings, loyalty benefits, travel-policy compliance in a single conversational turn), the authors demonstrate a dual-layer architecture: AgentCore Evaluations continuously scores live interactions with LLM-as-a-Judge across helpfulness, correctness, and goal completion, with reasoning attached to every score; on quality drops it runs pattern analysis over recent low-scoring sessions and recommends prompt, tool-selection, or orchestration fixes. AWS DevOps Agent acts as an on-call engineer, autonomously pulling CloudWatch logs, building resource topology graphs, correlating IAM policies and invocation traces to link an empty response to a missing permission.
The stack includes AgentCore runtime with built-in OpenTelemetry, the Strands Agents SDK (Swarm/Graph/Agents-as-Tools patterns), and the FAST fullstack template, forming a continuous monitor-analyze-improve-deploy feedback loop.
More from coding & agent
- Five teams, 72 hours, same Grok Bot: a controlled experiment to test Grok as a startup builder — tallmetommy · 2026-09-12
- Idempotent Webhook Handler Cheatsheet: Retries Are Normal, Double Work Is the Bug — blaizedsouza · 2026-09-12
- Arcee open-sources NAC agent harness; 489 tool calls, 67 passing tests on DeepSeek V4.1 Flash — MaziyarPanahi · 2026-09-12
- Tencent's Open-Source CubeSandbox v0.7 Ships 60ms-Cold-Start MicroVMs for Agents — dr_cintas · 2026-09-12
- DeepSeek V4.1 Flash tool calling: 489 calls, 56 min, 67 passing tests on LIGO data — MaziyarPanahi · 2026-09-12
- Infinitty Pro launches: metal-driven agent-first macOS terminal at $49 one-time — jasonkneen · 2026-09-12