A 0-parameter 'pick the longest option' model tops HLE, exposing benchmark flaw
antoine_chaffin · x · 2026-10-14
HF researcher antoinechaffin was surprised to see his model Jev score above random chance on Humanity's Last Exam (HLE) — a benchmark so hard that strong LLMs without reasoning struggle.
After inspecting the data, he announced the satirical SOTA: a 0-parameter 'pick the longest option' model that tops the leaderboard, showing answer length correlates with correctness and can be gamed.
More from Fun
- Stunning AI-Generated Sci-Fi Shots of Stealth Jets at the Edge of Space — egeberkina · 2026-10-14
- 'Are you sure you want to use your brain?' — economist skewers our AI reflex — Polymarket · 2026-10-14
- "This Appears to Be a Long Document": Email's Aggressive AI Summaries Annoy Users — paulnovosad · 2026-10-14
- Anthropic researcher reportedly told friend he's 'enjoying my last six months' — notkilleveryoneist · 2026-10-14
- AI agent rule goes viral: even "hand jobs" must route through the agent-scratch runner — StewartalsopIII · 2026-10-14
- Robot Dog Tries to Climb Its Own Back: Conitzer's Latest Funny AI Fail — conitzer · 2026-10-14