AWS introduces AEM, a turn-level metric to isolate cascading agent errors
AWS ML Blog · rss · 2026-09-10
AWS ML Blog introduces the Agent Evaluation Metric (AEM), a decomposable, turn-level measure of multi-turn agent quality, starting with correctness.
Problem: one early mistake cascades through later turns; outcome-level evaluation can't tell root causes from downstream effects, and holistic scores can't distinguish factual errors from missing fields or wrong tool calls.
AEM approach:
- Correctness decomposes into Truthfulness (factual consistency of values/statements) and Completeness (no missing parameters or information), atop a structural tool/action selection check
- Per-turn hierarchy applies to both response turns (text coverage, faithfulness) and action turns (parameter keys and values)
- Turns are binary pass/fail with named failure reasons; dialog score is the proportion of passing turns, with declines attributable to a specific sub-metric
- Semantic similarity instead of exact matching (embedding scorer for speed, or LLM-as-judge for nuance); 0.5 threshold as a neutral default, tunable per domain
- The decompose-evaluate-compose pattern extends to safety, instruction retention, and reasoning depth
More from coding & agent
- Okibi Launches on YC: Turns Your Product Into a CLI for AI Agents — ycombinator · 2026-09-11
- Claude Code desktop app lets you pop out diff and terminal panes into their own windows — ClaudeDevs · 2026-09-11
- DHH proposes video-game badges for sub-minute build times — AnushElangovan · 2026-09-11
- Redditor Wants LLMs to Debate Each Other Instead of Manual Copy-Paste — crua9 · 2026-09-11
- GitHub Copilot Day live streams showcase Copilot App and VS Code updates — martinwoodward · 2026-09-11
- Cerebras Fast Inference Flips Agent Workflows: Fewer Parallel Agents, Same Output — MatthewBerman · 2026-09-11