Hamel Husain's 11 Lessons for Building AI Products in the Agent Era: Read Traces First, Distrust Deliverables

Hugo Bowne-Anderson published his podcast with Hamel Husain, "The Rise of the AI Scientist," and distilled 11 lessons for building AI products in the agent era. The overall stance: AI accelerates building, but the responsibility to verify data and judge answers remains with humans. The thread opens with a telling case: a data-analysis agent returned a net revenue figure of $4.21 million without showing definitions, tables, filters, joins, dates, or fiscal-year conventions—exactly the problem this methodology addresses.

Confirmed

• Hamel's core advice: if you're not willing to personally verify your own data, don't build AI products; agents can surface traces but cannot supply the curiosity and judgment needed to understand them
• He argues the claim that "data science is dead" is wrong: AI creates more data, more traces, and more noisy signals requiring reasoning and debugging, making data-science judgment more valuable than ever
• The methodology starts with "forget evals for now": manually inspect 10 traces, and the terrible failures you dig up will tell you what evals should test; traces prove execution, not correctness
• Deliverables don't matter—Hamel says "the deliverable is just proof"; products should be built around the evidence experts need to judge whether AI answers are trustworthy
• Tools like Claude Code make it too easy to ship AI products users cannot verify, and evals cannot save bad product design; a product being hard to eval is itself a signal of poor design
• AI's value is helping experts miss less—proactively surfacing possible facts, contradictions, and open questions (even if partly wrong), while final sign-off and importance judgment remain human; a 52-page AI-generated report is mush for evaluation purposes and should be decomposed into fact-finding, contradiction detection, and open questions
• Product design determines what evaluation data you get: making outputs easy to verify lets the product automatically collect better signals—user feedback, annotations, and more classifiable errors
• Annotation should teach agents to proactively surface higher-value samples, making labeling faster and failure modes clearer

Why it matters

In an environment where agents produce results rapidly, these lessons offer a concrete defense: designing evals from manual trace inspection and centering product design on evidence rather than deliverables directly addresses the widespread risk of unverifiable AI-generated content. For data-science practitioners, it also rebuts the "data science is dead" anxiety, arguing that judgment and debugging skills are scarcer—and more valuable—in the agent era.

2026-09-04 ~ 2026-09-05 · 13 related posts

• [source] Agent Returns $4.21M Net Revenue With No Definitions: Hamel Husain's 11 Lessons for AI Products — hugobowne · 2026-09-04
• [source] A Trace Proves What Ran, Not That the Answer Is Right — hugobowne · 2026-09-04
• The Deliverable Is Just the Proof: Build AI Products Around Evidence — hugobowne · 2026-09-04
• Claude Code Makes It Easy to Ship AI Products Users Can't Verify — hugobowne · 2026-09-04
• AI Should Help Experts Miss Less, Not Replace Their Judgment — hugobowne · 2026-09-04
• A 52-Page AI Report Is Eval Mush: Decompose It Before Evaluating — hugobowne · 2026-09-04
• Your Product Design Determines the Eval Data You Get — hugobowne · 2026-09-04
• Hamel Husain: Labels Should Teach Your Agent What to Surface Next — hugobowne · 2026-09-04
• Forget Evals for a Second: Inspect 10 Traces First, Says Hamel Husain — hugobowne · 2026-09-04
• Hamel Husain: Data Science Isn't Dead — Agents Create More Signals to Debug, Not Less — hugobowne · 2026-09-04
• Don't Build AI Products If You Won't Inspect Your Data, Says Hamel Husain — hugobowne · 2026-09-04
• Hamel Husain: Don't Build AI Products If You Won't Inspect Your Data — hugobowne · 2026-09-04
• [source] Hamel Husain: Hard-to-eval products are bad products — and AI makes data science more valuable — hugobowne · 2026-09-05