Start evals from 100 production traces: a 7-step recipe from Hamel Husain's FAQ
HamelHusain · x · 2026-11-06
A shared practical recipe plus link to Hamel Husain & Shreya Shankar's AI Evals FAQ (built from teaching 5,000+ engineers and PMs).
Starting evals with traces but no test suite:
- Pull 100 recent production traces
- Read them and note failures in plain English
- Group notes into failure types
- Count each type
- Write pass/fail checks for the top 3
- Compare checks against human labels
- Add every new failure to the dataset
Key point: generic metrics can look healthy while what users care about is broken — build a test suite shaped like your real failures. The FAQ covers minimal viable eval setup, budget allocation, evaluating subjective/multi-step outputs, and cost control.
More from coding & agent
- Yukon syncs tables near-real-time at 1M events/sec, built so agents can query what changed — testingcatalog · 2026-11-06
- How one creator turned X posts into 40,000 email subscribers with a Go/No-Go AI agent — VibeMarketer_ · 2026-11-06
- Ripple-65B preview lands on Together AI: 80.5 on Terminal-Bench 2.1, 1M context — togethercompute · 2026-11-06
- Garry Tan: Capy's coordinating subagents boosted his PR output ~13x — garrytan · 2026-11-06
- GitHub Copilot's agent stack: cross-repo projects, parallel sessions, desktop app and automations — 0xkarasy · 2026-11-06
- Stripe opens its data to agents: free queries through Jan 2027, SQL shown with every answer — emilygsands · 2026-11-06