Hamel Husain: make AI outputs easier to evaluate by redesigning the product, not the evals

HamelHusain · x · 2026-09-26

Hamel Husain and Shreya Shankar answer "how can I make AI outputs easier for people to evaluate?" — start with product design, not the eval pipeline.

Surface intermediate outputs: instead of asking a doctor to critique a generated medical report, show the extracted facts with links to sources and let them correct facts or resolve conflicting evidence before generating the report — keeping the human in the loop and building trust
Reduce review friction: render outputs in familiar formats (emails as emails, syntax-highlighted code), keep reviewer context on one screen with collapsible details, add keyboard shortcuts for moving between examples and recording judgments, and show progress like "45 of 100 examples reviewed"

The companion post "It's Hard to Eval" Is a Product Smell expands on this with before-and-after mockups.

More from coding & agent