A 0-parameter "pick the longest option" model beats strong LLMs on Humanity's Last Exam
tomaarsen · x · 2026-10-14
Antoine Chaffin found that a model called "Jev" scored above random chance on Humanity's Last Exam (HLE) — a benchmark that's hard even for strong LLMs without reasoning. Inspecting the data, he announced a "SOTA decision model" topping the benchmark with… 0 parameters: it just picks the longest option.
This suggests HLE multiple-choice items may be biased toward longer correct answers, letting a length heuristic beat random without any knowledge. Reposter tomaarsen called it "so sad" — yet another benchmark contamination/bias alarm.
More from Fun
- User Asks Grok to Count Pints in a Chart, Gets 34⅓ Pints of Guinness — AI_Andrew · 2026-10-14
- NYT reporter interviews live Chinese AI agent fleet traced to Tencent — dylfreed · 2026-10-14
- MBA Redefined: Master of Bot Administration in the Agent Era — yunta_tsai · 2026-10-14
- Explaining how diffusion models work, using beavers — manicman1999 · 2026-10-14
- Open-source Screen Studio Clone Sparks Fight Over Missing Download Button — Rasmic · 2026-10-14
- Luke Wroblewski charts 19 years of iOS calculator design evolution — LukeW · 2026-10-14