Hamel Husain and Shreya Shankar: never skip error discovery when building AI evals
FinanceYF5 · x · 2026-09-23
Lenny's Newsletter published an advanced evals article by Hamel Husain and Shreya Shankar. Drawing from work with 50+ AI companies, they argue the most common mistake is jumping straight to writing metrics: logs are painful to review and metrics are easy to automate, so teams end up measuring the wrong things. Error discovery should be treated like product research—find the problems worth solving before deciding what to test. The piece also covers which steps can (and cannot) be automated, and a free plugin that lets a coding agent do most of the heavy lifting, plus a related course for engineers and PMs.
Related event: Husain and Shankar Publish Deep Guide on AI Evals(4 posts)→
More from coding & agent
- How to cut Fable 5.2 token costs by 90%: Arc routing plus a lean CLAUDE.md — EXM7777 · 2026-11-12
- Hamel Husain's complete AI evals framework for PMs and engineers — aakashgupta · 2026-11-12
- AI coding in one year: from unusable to full automation with no code review — angelaqdai · 2026-11-12
- Interactions Tool Searches Web Animations by Motion for Claude Code to Rebuild — yuwen_lu_ · 2026-11-12
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Senate Bill Would Force Big Platforms to Accept AI Agents, Raising Auth Privacy Questions — Prof_CarlaReyes · 2026-11-12