LLM Leaderboards Manipulated by Eval Config: Same Model Scores 31% to 89%
rohanpaul_ai · x · 2026-09-01
A study shows LLM leaderboard results are highly sensitive to evaluation configuration. Keeping models and 3,679 questions fixed, changing only prompt format, option order, and scoring method caused gemma4-31b to score between 31% and 89%, and 4 of 12 models reached rank 1 under some configuration. 95.7% of average gaps between neighboring models come from config-fragile items. The biggest instability source is scoring method: generating answers vs. choosing highest-likelihood option.
More from Models
- Open source is so back: dozens of open models dropped within 48 hours — beffjezos · 2026-10-16
- Gemini users hit usage caps on 3.1 Pro and 4 Argon, resets not until Oct 16 — R3d-Gr33n-Blu3 · 2026-10-16
- Gemini desktop app can't accept images — an image model that can't take images — Rebl · 2026-10-16
- Viral screenshot claims GPT-6 Astra proved a physics benchmark wrong, lifting its score to 98% — Confident_Salt_8108 · 2026-10-16
- Apollo Research: GLM-5.3 unprompted reasons about its actions becoming training data — _lewtun · 2026-10-16
- Opus 5.5 turns heads with surprisingly polished vertical video demo — FinanceYF5 · 2026-10-16