EMNLP paper: tweaking multiple-choice scoring schemes reshuffles LLM leaderboards
boydgraber · x · 2026-10-26
Nishant Balepur et al. (UMD) present "Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation" at EMNLP, showing that scoring schemes alone can reshape benchmark rankings.
- Premise: small research tweaks to how multiple-choice answers are scored can make benchmarks test model abilities far better, without touching prompts or metrics
- Takeaway 1: schemes disrupt leaderboard ranks more than they reward — partial-credit/abstention schemes beat plain number-right prompts
- Takeaway 2: Answer-Until-Correct best mirrors LLM-Arena rankings, suggesting models should be asked to yield scores others agree with
- Takeaway 3: schemes reveal LLM quirks — GPT is overconfident, Command R+ abstains, LLaMA won't disambiguate
More from Models
- Ruff creator Charlie Marsh: frontier models find infinite bugs in any software — charliermarsh · 2026-10-26
- Dev testing one skill across Opus 5.5 and Sol 6.1 finds instruction following 'all over the map' — lucasmeijer · 2026-10-26
- supermemory unveils learner-1, claiming continual learning for any model — julianweisser · 2026-10-26
- Anthropic says Opus 5.5 cuts cost per task by ~40% vs Opus 5 — taufiqintech · 2026-10-26
- AI search APIs ranked: top 12 within 7 points, yet prices differ nearly 7x — FinanceYF5 · 2026-10-26
- OpenAI Dots coming to EU/EEA and UK in a few weeks, slides confirm — testingcatalog · 2026-10-26