Hamel Husain: public benchmark quality is abysmal — broken graders, sloppy harnesses
HamelHusain · x · 2026-10-21
After 48 straight hours reviewing public benchmark traces, Hamel Husain reports the quality is abysmal: misconfigured graders, sloppy harnesses that ding correct agents, half-specified environments with no error feedback so agents can't self-correct, flat-out wrong assertions, and agents not given sensible tools or proper API usage. He credits @xeophon with months of deep work on this. Key implication: some low agent benchmark scores may reflect benchmark bugs, not model failures.
More from coding & agent
- Community builds 4 GUIs for the Pi coding agent as official client stalls — solyarisoftware · 2026-10-21
- Nace.AI Launches Drex Max 2.0, a 27B Local Decision Model for Computer Use — DotaMate · 2026-10-21
- Musk shares how Grok Bot cleared 204 support tickets in ~10 minutes with one engineer approval — elonmusk · 2026-10-21
- pstack ships meta-skill that has your agent draw statecharts, then find bugs via TLA+ formal verification — DavidKPiano · 2026-10-21
- Study: decision-model controllers raise agent cost 19% with zero success gain — Naman Vats · 2026-10-21
- Tencent's RiseHarness iteratively evolves agent harnesses without touching model weights — Tencent-Hunyuan · 2026-10-21