Hamel Husain: Firms should pick models with private evals, not public benchmarks

On November 9, in a discussion around the "evaluation military-industrial complex," well-known evaluation expert Hamel Husain made a core argument: the industry over-relies on third-party benchmarks to decide which model to adopt, when the right approach is to run more private evaluations tailored to one's own product and specific tasks, using self-built evals as the primary basis for decision-making.

Confirmed

• Hamel Husain believes public benchmarks are too generic — more valuable to large labs and researchers, and merely indicative for ordinary users. The proper division of labor is that public benchmarks only serve the top of the evaluation funnel — deciding "whether to even consider this new model" — while subsequent filtering is left to private evals.
• xeophon proposed an isolation principle for internal evaluations: those responsible for training should not be able to access internal benchmarks, or even know what's in them; the training side gets only an endpoint and a score.
• Megan Risdal added that to prevent internal benchmarks from being overfit, the core design of a hill-climbing-resistant evaluation setup is a continuous, constantly evolving stream of entirely new tasks rather than a fixed question bank, and the interpretation of Goodhart's law must adapt accordingly.
• xeophon and llmwizard noted that benchmark collections like the Artificial Analysis Intelligence Index frequently rotate their underlying benchmarks largely because scores get targeted overfitting (Goodharting); llmwizard went further, arguing these leaderboards have become marketing tools for the labs.
• mertgulsun responded to Hamel Husain that the two are complementary: developers should maintain their own evals for their needs while also relying on third-party evaluations.

Why it matters

• The discussion touches on a methodological divide in today's AI industry over model selection and evaluation: public scores are easily gamed or even manipulated, while performance on real enterprise tasks correlates only loosely with generic benchmarks.
• The isolation design of internal evals and the continuous task-rotation mechanism offer actionable engineering practices for keeping evaluations from being polluted by "hill climbing."

2026-11-09 ~ 2026-11-09 · 7 related posts

• [source] Hamel Husain: companies should rely on private evals, not third-party benchmarks, to pick models — HamelHusain · 2026-11-09
• Private evals vs third-party benchmarks: Hamel Husain sparks debate on model selection — mert_gulsun · 2026-11-09
• [source] Public benchmarks are top-of-funnel only: Hamel Husain on building enterprise evals — HamelHusain · 2026-11-09
• Benchmark indexes rotate constantly due to Goodharting, and mostly serve as lab marketing — llm_wizard · 2026-11-09
• Leaderboards keep rotating benchmarks because of goodharting, argues xeophon — xeophon · 2026-11-09
• [source] Evolving streams of novel tasks are core to hillclimbing evals, says Megan Risdal — MeganRisdal · 2026-11-09
• Don't let the training team see your internal benchmarks — just return a score — xeophon · 2026-11-09