Hamel Husain: sell evals to your team with error analysis, not abstract pitches
HamelHusain · x · 2026-10-09
Hamel Husain and Shreya Shankar answer how to make the case for evals at your team: don't sell the concept, show the data.
- Do the error analysis yourself: review 50-100 real user conversations and identify the most common failure modes
- Present top failure modes, frequency metrics for high-impact errors, and surprising user behaviors
- Frame fixed bugs as "prevented production issues" and keep a running log of errors caught, fixes, and avoided impact, shared weekly or monthly
- Concrete numbers like "we caught 47 issues before users saw them" beat abstract pitches; show error rates drop after fixes
Key idea: treat evaluation as part of development, and let results lead the conversation.
More from coding & agent
- Claude cut a 2.6-hour video export to 20 minutes — but only after being told to rebuild the server — byhewar · 2026-10-23
- 3 flight-booking rules my AI agent checks that I'd never do by hand — Efistoffeles · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23
- How do you audit AI agent decisions in regulated workflows? A field survey — Ok_Way_1993 · 2026-10-23