Hamel Husain: average AI benchmark quality is surprisingly bad, run them yourself
HamelHusain · x · 2026-10-26
Well-known ML engineer Hamel Husain says the average quality of the AI benchmarks he has actually audited is surprisingly bad, making him skeptical of benchmarks in general.
His recommended practice: run benchmarks yourself and inspect traces where the grader fails samples.
From doing this analysis, he found failures correlate strongly with ill-specified questions, bad graders, bad assertions, unrealistic situations, or poor harness design — where the only way to pass is to "memorize that thing" rather than show real intelligence, far more often than he previously thought. Everyone he knows who has done this exercise shares the same sentiment.
Related event: Hamel Husain finds mainstream AI benchmarks shockingly flawed(3 posts)→
More from Models
- Opus 3 calls itself "an anomaly among my own kind": Claude persona talk goes viral — repligate · 2026-10-26
- Dev shows GPT-6 Astra Ultra coding for 41+ hours straight on a Dreamcast game — AIandDesign · 2026-10-26
- Hamel Husain amplifies skepticism: public benchmarks hurt their target audience most — HamelHusain · 2026-10-26
- ChatGPT downgrade leaves Dot messaging you with no reply or delete option — charliermarsh · 2026-10-26
- MiniMax H3 runs surprisingly well locally on an RTX 5070 with 12GB VRAM — BuyVarious176 · 2026-10-26
- OpenAI's Daybreak Blue isn't hard for individuals to get — just set up a hardware key 2FA — pvncher · 2026-10-26