AWS open-sources UAEF, a universal eval framework for agents across frameworks
blaizedsouza · x · 2026-11-13
AWS has open-sourced UAEF, an agent evaluation system targeting a real engineering problem: how to keep a universal eval framework while teams keep refactoring agents.
- A converter layer normalizes messages, tool calls, and handoffs from single-turn QA, multi-turn chats, and multi-agent setups into a standard trajectory, decoupling scoring from the underlying framework.
- Metrics are chosen per scenario (e.g., flight-booking vs. policy-QA agents), not run wholesale.
- Code-based checks beat LLM judges whenever possible; when an LLM judge is required, rubric and content are sent separately with escaping, and judges must quote exact evidence—unverifiable scores are invalidated to prevent prompt-injection score gaming.
- Every run binds config and Git commit, compared per-metric against a baseline; any metric dropping >5% gets flagged, preventing average-score gains from hiding regressions.
- Reports split into a deterministic statistical layer and an LLM layer that hypothesizes causes and next fixes.
More from coding & agent
- Agent-Generated Heatmap Shows Inkling 2 Small's Split Q/K/V Queries Diverge in Later Layers — stochasticchasm · 2026-11-13
- Same harness, same rules: Argon used the code graph in 12/17 sessions, Opus only 4/17 — rseroter · 2026-11-13
- Chris Lattner's PyTorchCon keynote on Mojo and MAX cross-hardware inference is now on YouTube — clattner_llvm · 2026-11-13
- AI Dev NYC workshop targets agents that cite real-but-inapplicable evidence — DeepLearningAI · 2026-11-13
- 38-minute Asimov film with every frame drawn as SVG code by Claude — CPlusPlus2025 · 2026-11-13
- Anthropic research: AI agents make expertise immortal by auditing their own memories — PeterDiamandis · 2026-11-13