AWS shows how to plug calibrated decision models into Bedrock AgentCore Evaluations
AWS ML Blog · rss · 2026-11-05
An AWS ML Blog tutorial shows how to connect decision models — small specialized models that output calibrated probability distributions over fixed answer sets — to Amazon Bedrock AgentCore Evaluations as a deterministic scoring layer alongside LLM-as-a-judge.
- Concept: Decision models suit narrow, repeatable checks (policy compliance, tool-call safety) where you need thresholdable, programmable confidence scores. Two backends are covered: TypeSafe's hosted Jev and the open-source 2B-parameter Strands Decider 2B, both supporting yes/no (noul), multi-choice (up to 255 options), and 2-10 level score questions.
- Division of labor: Decision models act as a screening layer passing fail signals across metrics; turns flagged for deeper review go to built-in LLM evaluators whose explanations aid debugging.
- Hands-on demo: An HR Assistant agent built with Strands Agents and five mock-data tools is deployed to AgentCore runtime; two Python scripts deploy deterministic Lambda evaluators (length checks, regex fact-checking), register code-based evaluators, run on-demand evaluation over five labeled scenarios, and configure online evaluation with 100% sampling. Full code is in the AgentCore samples GitHub repo.
More from coding & agent
- Claude hand-writes a legal-citation detector net, 1.8M cites/s on Apple ANE — paul_cal · 2026-11-05
- Mistral's Vibe CLI releases version 3.1 — qtnx_ · 2026-11-05
- Applied Compute's Ari Autoresearch lifts Qwen3.5-9B chess reward 5.3x in a 16-hour autonomous run — ypatil125 · 2026-11-05
- Wattly MCP lets ChatGPT, Claude and Grok control your Tesla — Scobleizer · 2026-11-05
- Self-hosted AI platform: 3 old GPUs, vLLM per card, and a semantic router as one endpoint — JeremyCMorgan · 2026-11-05
- Sketch to printable STL in 4.5 minutes: an open-source Kimi K3.1 Pro CAD workflow — Arindam_1729 · 2026-11-05