Hamel Husain on the Evals-Industrial Complex: benchmarks die once people benchmaxx them
HamelHusain · x · 2026-11-11
Reacting to Claude scoring below Fable 5.1 and Sonnet on Harvey's legal benchmarks, Hamel Husain coins the "Evals-Industrial complex": if you can't convince people to benchmaxx on your benchmark it's deemed worthless, but if they do benchmaxx on it, it statistically becomes worthless anyway.
More from Models
- Alleged Gemini 4 Carbon builds 92-part 3D Paris physics lab from one image (unverified) — theREALmarvin · 2026-11-11
- GBENCH: OpenAI's GPT-6 Astra tops unsaturated coding benchmark with 83.3, sweeping top 3 — teortaxesTex · 2026-11-11
- Artificial Analysis launches Intelligence Index v5 at San Francisco event — ArtificialAnlys · 2026-11-11
- Fable 5.2 video generation via code appears to lack nudity filters in JS — mickeyxfriedman · 2026-11-11
- Lenny: Designers name Claude most often, Anthropic winning hearts and minds — lennysan · 2026-11-11
- LLMs Aren't Fundamentally Discrete: Tokens Are Just Readouts of a Continuous Process — Sauers_ · 2026-11-11