Air Canada's agent testing playbook: write the answer first when no benchmark exists
AI Engineer · youtube · 2026-10-22
On the AI Engineer channel, Nathanael Weill, who leads AI agent development at Air Canada, explains how to test agents when tasks are too domain-specific for public benchmarks and experts have no time to grade.
Core method — write the answer first:
- Start from a known answer, generate the input from it, and check whether the agent finds its way back
- This took a telecom RAG project from one expert-graded test per week to dozens of automated tests per day
For a supplier-contract review agent, he shows five ways to build a test set:
- Past real decisions
- Planted rule violations
- Removed clauses (does the agent notice?)
- Changes that shouldn't matter (negative samples)
- Contract templates with filled-in blanks
Another key claim: avoid LLM judges whenever code can check the answer; expert feedback should go into building better test sets rather than per-case grading.
More from coding & agent
- Longman Dictionary MCP Server Serves Structured Definitions to AI Agents — modelcontextprotocol · 2026-10-22
- Law professor rethinks Microsoft Copilot after hands-on work inside a major law firm — jkubicki · 2026-10-22
- A complete guide to giving Claude perfect memory with folders, CLAUDE.md and daily logs — evielync · 2026-10-22
- Dev shows off self-built infra stack — HTTP clients, attention kernels, model routers — to quit dependencies — wavefnx · 2026-10-22
- Codex app reportedly ignores model selection, randomly switching to other models — zeeg · 2026-10-22
- Anthropic's /claude-api prompt-audit cuts agent costs 9% and lifts accuracy ~2 points by fixing CLAUDE.md — Arindam_1729 · 2026-10-22