0-Parameter 'Pick the Longest Answer' Heuristic Tops HLE Decision Benchmark at 20.7%
antoine_chaffin · x · 2026-10-14
Antoine Chaffin measured Jev at 19.7% on HLE's 513 text-only multiple-choice questions (the set behind the AA Decision Index). The twist: always picking the longest option scores 20.7% — above random guessing (16.4%) and every decision model tested, an effect driven mostly by Math and Physics questions.
He stresses no model systematically prefers long answers; rather, with 513 questions, a few points of gap is within noise. Removing length-advantaged questions brings all models closer to random guessing with none standing out.
His takeaway: always inspect the data and run dumb baselines — eval sets are full of length/position biases, and a 0-parameter heuristic topping the leaderboard means the benchmark is mostly measuring noise and dataset artifacts, not knowledge.
More from Models
- Dev Pays for Gemini Ultra, Walks Away Unimpressed by Gemini 4 Argon + Antigravity — yihui_indie · 2026-10-14
- NVIDIA Open-Sources a Chest CT Model Trained on 70,111 Scans That Writes Reports in Under 30 Seconds — MaziyarPanahi · 2026-10-14
- Gemini 4 Argon first impressions: superb at code analysis, but quotas burn out in five hours — CarlosEduardoAraujo · 2026-10-14
- How to access Gemini 4 Argon: App, AI Studio and Antigravity, from $19.99/month — TawohAwa · 2026-10-14
- Unverified leaks compare Claude Opus 5.5, Gemini 4 Argon and GPT-6 Astra in early tests — TawohAwa · 2026-10-14
- Meta launches coding agent Muse Wonder Full at $50 or $200 per month — lemire · 2026-10-14