AI Evals 101: Building your first agent eval with LLM judges and error analysis
Prompt Engineering · youtube · 2026-11-06
A hands-on tutorial video on AI evals: why testing agents differs from testing traditional software, the three grader types (code checks, LLM judge, human review), and a full walkthrough building an eval set for a text-to-SQL agent — using error analysis to find real failures, calibrating an LLM judge against human labels, then fixing the agent and rerunning evals. Includes further reading from Anthropic's evals guide, Hamel Husain's AI Evals FAQ, and a paper on annotation errors breaking text-to-SQL benchmarks.
More from coding & agent
- Declarative Software Factory: making 'Done' explicit as agents do the work — hwchase17 · 2026-11-23
- huashu-design 3.5 ships: WeChat mini-program design support, two days after 3.4 — AlchainHust · 2026-11-23
- Dev Fixes 3 Agent Eval Holes: Blind Recount Hits 89% Agreement, Pooled 4% Miss Rate Hid a Bad Lane — armanidev_ · 2026-11-23
- Sharing one memory across Claude Code, Codex and local models over MCP: what worked — JoeBro121211 · 2026-11-23
- Firecrawl vs context.dev: Developers weigh web data stacks for AI agents — lewishorsley96 · 2026-11-23
- Popfold: an open-source three.js engine built with parallel Claude Code sessions — cherichita · 2026-11-23