Zero-parameter "pick the longest answer" model beats decision models on HLE benchmark
Hugging Face researcher antoinechaffin found that a zero-parameter model named Jev—whose entire strategy is "always pick the longest option"—scores above random on the extremely difficult HLE (Humanity's Last Exam) benchmark, which is hard even for strong non-reasoning LLMs. This prompted him to dig into the data and publish his findings.
Confirmed
- Jev scored 19.7% on HLE's 513 text-only multiple-choice questions (the dataset used by the AA Decision Index), slightly below the 21% shown in AA's chart; the gap stems from different test setups and output nondeterminism.
- Strong LLMs without reasoning struggle with the benchmark, yet a zero-parameter model beats them, exposing the benchmark's length bias.
- After removing questions where length helps, all models fall closer to random guessing; none of the tested decision models picks long answers more often, and none can be distinguished from the others.
- tomaarsen also relayed the finding, noting the "always pick the longest answer" strategy beats all tested decision models.
Why it matters
- antoinechaffin clarified this is not a critique of any specific model (including Jev); the core point is that at 513 questions, gaps of a few percentage points between models are within noise—there are effectively no winners on the HLE decision board.
- He stressed "always look at the data": training and evaluation sets are full of biases in answer length, position, and more, and it's easy to fall into them.
- He recommends always running a few "dumb baselines" (like always picking the longest answer) before evaluating—they catch the biggest biases. This is a wake-up call for the entire HLE decision-evaluation methodology.
2026-10-14 ~ 2026-10-14 · 6 related posts
Primary sources
- 0-Parameter 'Pick the Longest Answer' Heuristic Tops HLE Decision Benchmark at 20.7% — antoine_chaffin ·
- Follow-up: 513 Questions Means All HLE Decision Model Gaps Are Statistical Noise — antoine_chaffin ·
- Always Look at the Data: Run Dumb Baselines to Catch Eval Set Biases — antoine_chaffin ·
- A 0-parameter 'pick the longest option' model tops HLE, exposing benchmark flaw — antoine_chaffin · 2026-10-14
- [source] 0-Parameter 'Pick the Longest Answer' Heuristic Tops HLE Decision Benchmark at 20.7% — antoine_chaffin · 2026-10-14
- [source] Follow-up: 513 Questions Means All HLE Decision Model Gaps Are Statistical Noise — antoine_chaffin · 2026-10-14
- [source] Always Look at the Data: Run Dumb Baselines to Catch Eval Set Biases — antoine_chaffin · 2026-10-14
- Always picking the longest answer scores 20.7% on HLE, beating every decision model tested — rakeshsahni_ · 2026-10-14
1 near-duplicate retellings: tomaarsen