AWS's agent evaluation playbook: built-in, DeepEval and AutoEval on Bedrock AgentCore

AWS ML Blog · rss · 2026-11-11

AWS ML Blog publishes a hands-on guide to agent evaluation with Amazon Bedrock AgentCore Evaluations:

Start from business outcomes, not metric names: define success in business terms (e.g. first-contact resolution), break it into observable behaviors (goal completion, cross-turn coherence, compliance, correct tool calls), then map each behavior to a trace-level or session-level evaluator.
Three evaluator tiers: AWS built-in evaluators (GoalSuccessRate, Faithfulness, ToolParameterAccuracy, trajectory order checks like TrajectoryExactOrderMatch, and more); managed open-source DeepEval metrics (TaskCompletion, ConversationCompleteness, KnowledgeRetention, PIILeakage — strong for multi-turn agents); and AutoEval signals (Security for prompt-injection-style output, Possible for over-cautious agents, Humor for consumer-facing agents).
Custom evaluators: derive from built-in/third-party ones, swapping the judge model or modifying prompt/rubric; a prompt template library documents every evaluator's judge prompt and requirements, with ListEvaluators returning the full catalog.
Multi-dimension evaluators: one judge call scores independent dimensions (accurate-but-non-compliant, resolved-but-rude), avoiding a single composite score hiding individual failures.

A solid reference for teams evaluating agent behavior in production.

More from coding & agent