Hamel Husain: BenchGate is pervasive — look at failing traces in any public benchmark
HamelHusain · x · 2026-10-21
AI engineer Hamel Husain warns that 'BenchGate' — quality problems in public LLM benchmarks — is still pervasive: pull up a handful of traces from failing tests in almost any public benchmark and you'll be shocked. Quoting a tool for browsing MMLU, he doubts people would 'knife fight for a percent' on this benchmark if its actual contents were widely known.
More from Research
- Study: decision-model controllers raise agent cost 19% with zero success gain — Naman Vats · 2026-10-21
- Tencent's RiseHarness iteratively evolves agent harnesses without touching model weights — Tencent-Hunyuan · 2026-10-21
- More memory, worse agents: Anti-Persistence discards synthesized guidance to cut errors — IUB · 2026-10-21
- CFG lifts single-step discrete action success from 27.1% to 49.2% in robot policy training — YouJiacheng · 2026-10-21
- Menlo Research open-sources Cyclotron, an RL training framework for its Asimov humanoid robot — freelerobot · 2026-10-21
- Father-son duo uses Hark to semi-autonomously medal on ARC Prize after days of self-revision — Deep-Firefighter-279 · 2026-10-21