Public benchmarks are top-of-funnel only: Hamel Husain on building enterprise evals
HamelHusain · x · 2026-11-09
Hamel Husain lays out his eval funnel: businesses should always maintain their own evals and base decisions primarily on them. Public benchmarks are too general — useful mainly to big labs — and should only serve top-of-funnel questions like "should I consider this new model?" Private evals should answer whether a model actually works for your workflow and drives business impact. He argues the current over-reliance on benchmarks is a marketing problem perpetuated by those who profit from them, while acknowledging evals are time-consuming.
More from coding & agent
- Agent harness evolution study: all 16 evolved agents trade old capabilities for new ones — ustc-community · 2026-11-09
- Researcher's open-source paper-review skill flagged ~30 errors in one manuscript proof — RobbWiller · 2026-11-09
- ezyang: fancier specification languages quietly grow your trusted computing base — ezyang · 2026-11-09
- Dev renders stunning animation in pure Python with Opus 5.5, zero 3D assets — Eric520CC · 2026-11-09
- REA Author Open-Sources 20+ Repos: An Agent-Powered Reverse Engineering Studio From Apps to Binaries — nikola_mr64990 · 2026-11-09
- Agent autonomously trained and deployed a local labeling model while owner was on vacation — max_paperclips · 2026-11-09