Hamel Husain: companies should rely on private evals, not third-party benchmarks, to pick models

HamelHusain · x · 2026-11-09

Evals expert Hamel Husain argues people rely far too heavily on third parties to decide which model to choose, and should instead run more product-specific private evals on their own tasks. He notes the benchmark ecosystem is skewed: public benchmarks mostly benefit big labs, and those profiting from benchmarks mislead consumers — a marketing and communications problem.

Original post →

Hamel Husain: companies should rely on private evals, not third-party benchmarks, to pick models

More from coding & agent