AWS Adds Skill-Level Agent Evals for Wrong-Skill and Skipped-Step Failures
AWS ML Blog · rss · 2026-09-23
AWS ML Blog details how to evaluate skill-equipped agents with the Strands Evals SDK and Amazon Bedrock AgentCore Evaluations. Skills follow the open Agent Skills standard and are reusable instruction sets (usually a SKILL.md) bundling instructions, tool bindings (APIs, MCP servers, local commands), knowledge, workflow and guardrails, loaded at runtime instead of cramming every procedure into a system prompt or fine-tuning the model.
Skill composability introduces two failure modes that generic output-quality metrics miss: invoking an inappropriate skill, and invoking the right skill but skipping or only partially following its steps. Both can yield fluent, plausible answers that never used your domain knowledge, so evaluating the final response alone is insufficient.
Three evaluators are added:
- Skill Selection Accuracy: binary, per invoked skill, judging whether the choice fit the task.
- Skill Instruction Following: five-level rating with evidence, per prescribed step.
- Skill Invoked (Strands Evals only): deterministic check that a named skill loaded, no model call.
The failures need different fixes: bad selection often means overlapping or ambiguous skill descriptions; incomplete following may need clearer steps, a different skill structure, or a stronger model. Per-skill results keep multi-skill runs diagnosable; if no skill is invoked, judge-based evaluators return no score, so pair them with SkillInvoked for regression tests with known routing requirements.
An HR assistant example (PTO planning vs benefits skills) illustrates the approach. Prerequisites: Python 3.10+, Bedrock InvokeModel permission, strands-agents-evals and strands-agents installed; on AgentCore, observability to CloudWatch and Transaction Search enabled, driven via the AgentCore CLI. Skill extraction supports the Strands AgentSkills plugin, Claude Code, Claude Agent SDK, OpenAI Agents SDK, Codex, Gemini CLI, OpenHands, Google ADK and generic SKILL.md reads.
More from coding & agent
- PrimeScientist uses adaptive MCTS to budget research agents: +10.3% reward with 50.6% fewer runs — dair_ai · 2026-09-23
- Dev's 3D Coding Harness Finds a Sharp Threshold: Strong Models Eat Scaffolding — rms80 · 2026-09-23
- ML-Intern in HuggingChat Ran the Whole Distillation Pipeline End to End — Gradio · 2026-09-23
- Gradio distills Qwen's 9B prompt rewriter into an 812MB 0.8B model that fits a laptop — Gradio · 2026-09-23
- OpenAI launches GPT-6 Sol and Luna, permanently cuts API prices 50% — aziz4ai · 2026-09-23
- Claude Opus 5.5 generates a 10v10 Halo-style multiplayer game from one prompt — Scobleizer · 2026-09-23