Hamel Husain Pokes Holes in Anthropic's New Evals Guide: Look at Data Before Writing Evals
HamelHusain · x · 2026-09-29
Hamel Husain put Anthropic's newly released evals workflow guide through its paces and flagged three flaws:
- Wrong order: The workflow creates evals before looking at data, then asks users to correct mistakes. He argues you should inspect data first to build understanding and prioritize before writing any evals.
- Crude annotation: Instead of generating markdown files where users "tell it" which labels are off, he says you should build your own annotation app — you're using coding agents, after all (as he did in a livestream).
- Overly broad evaluators: The scoped evaluators bundle too many failure types at once, which looking at data first would avoid.
He worries the guide could steer beginners toward bad eval-engineering habits, and shared notes for those who don't have time to watch the full walkthrough.
Related event: Anthropic's New Evals Guide Draws Scrutiny from Hamel Husain(3 posts)→
More from coding & agent
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- ffmpeg-skill gives coding agents local video editing via 42 FFmpeg tools and MCP — kajisho5 · 2026-11-16
- Every's Frontier team on Claude Fable 5.2: overnight agent jobs, cost tradeoffs, real builds — every · 2026-11-16
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16
- DeepSeek open-sources its app frontend "chat-demo", already at 3.9k stars — NielsRogge · 2026-11-16