Hamel Husain: public AI benchmarks are so broken that hill climbing means memorizing answers
HamelHusain · x · 2026-11-02
Eccentric's Hamel Husain warns that companies publishing benchmarks and selling consulting around them should be scrutinized. Running any public benchmark and inspecting failing traces typically reveals: tests too narrow (grader fails correct models), tests too broad (ambiguous criteria), poor harnesses, ground truth generated by another model introducing bias, and grading answer format instead of content. His conclusion: the only reasonable way to hill climb under such benchmarks is to memorize answers — and this is just the tip of the iceberg, with vendors building marketing on these low-quality evaluations.
Related event: Hamel Husain Slams Public AI Benchmarks as Rating-Agency-Style Business(4 posts)→
More from Models
- Grok Bot pricing debate: users say free tier makes no sense, back gifting instead — RachelVT42 · 2026-11-02
- Grok users hit weekly limits early as OpenAI doubles Pro caps — how do teams plan? — Entire_Struggle_2236 · 2026-11-02
- PyTorch interview question: model.eval() vs torch.no_grad() explained — cneuralnetwork · 2026-11-02
- Reflection to host Reddit AMA on Nov 3 about Beam, its first open-weight model — Aggressive-Sign-8167 · 2026-11-02
- Cursor Pro+ user burns through all quotas, says forced to upgrade to $200 Ultra plan — pramodk73 · 2026-11-02
- User regrets $500 OpenAI plan: 'ultra fast' is actually buggy computer use — imjustnewatai · 2026-11-02