AWS Introduces UAEF: Framework-Agnostic Agent Evaluation With 36 Metrics Across 7 Dimensions

AWS ML Blog · rss · 2026-11-11

AWS ML Blog introduces UAEF (Universal Agent Evaluation Framework), a framework-agnostic closed-loop evaluation system for AI agents, already used by 12 organizations in telecom, healthcare, and financial services.

Design

Adapters: normalize execution traces from LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore into a canonical AgentTrace—normalize first, score second—so metrics and comparisons stay consistent across frameworks and runtimes (local, HTTP endpoint, or managed).
Evaluation engine: 36 built-in metrics across 7 dimensions (tool calling, response quality, responsible AI, performance, multi-turn, multi-agent coordination, reasoning), each returning a [0,1] score; deterministic metrics plus LLM-as-a-judge metrics on Bedrock, with RAGAS/DeepEval integrations and custom metrics supported.
Judge hardening: agent output fed to judges is treated as untrusted input, escaped and sent separately from the rubric; each judge response must cite a verbatim span or the score is rejected; judge scores are evaluation signal only, not security controls (use Bedrock Guardrails for runtime enforcement).
Closed loop: PII screening before storage, experiment metadata in DynamoDB and full results in S3, version-vs-baseline regression comparison, and reports guiding the next prompt/model/tool iteration.

Includes a playbook for metric selection and dataset prep, plus runnable notebooks per framework.

Original post →

AWS Introduces UAEF: Framework-Agnostic Agent Evaluation With 36 Metrics Across 7 Dimensions

More from coding & agent