Hamel Husain: Benchmarks are used like credit scores, so criticism is warranted

HamelHusain · x · 2026-11-05

Hamel Husain argues the analogy between open source software and AI benchmarks is a false equivalence: OSS outputs aren't consumed directly for decisions, while benchmark scores are taken at face value like Yelp ratings or credit scores.

Benchmarks come with no disclaimers, most consumers lack the data skills to inspect traces and see their brittleness, and even skilled people rarely question them—often being shocked when they finally look at the traces. He says the biggest issue is the large dissonance between benchmark marketing and what they actually measure, so instead of deflecting criticism, the communication around benchmarks needs to be fixed.

Original post →

Hamel Husain: Benchmarks are used like credit scores, so criticism is warranted

More from Models