Always picking the longest answer scores 20.7% on HLE, beating every decision model tested
rakeshsahni_ · x · 2026-10-14
rakeshsahni measured Jev at 19.7% on HLE's 513 text-only multiple-choice questions (the set used in the AA Decision Index), slightly below the AA chart's 21% due to setup differences and non-deterministic outputs.
On the same set, the naive strategy of always picking the longest answer scores 20.7% versus 16.4% for random guessing — above every decision model tested. The effect comes mostly from math and physics questions, suggesting answer-length bias inflates model scores. Jev's creator antoinechaffin replied that his SOTA claim was sarcastic.