Hamel Husain: refresh stale eval datasets with regular error analysis, retire passing ones

HamelHusain · x · 2026-09-23

In their AI Evals FAQ, Hamel Husain and Shreya Shankar answer what to do when your gold eval dataset goes stale:

Evals naturally go stale as products and users change; use regular error analysis to find new problems and update examples/reference answers, with review cadence tied to how fast your product evolves
Evals catch regressions like unit tests but cost far more to maintain — weigh cost vs signal value; if everything keeps passing, retire or run the eval less often
Scores losing comparability with old runs is fine: evals exist to provide challenges you can hill-climb against
For long-horizon progress tracking, lean on product metrics (churn, active users, revenue) alongside eval scores

More from coding & agent