EMNLP 2026 paper shows alternative scoring schemes reshuffle LLM leaderboard ranks
BrihiJ · x · 2026-10-20
A University of Maryland / NYU / NTU paper at EMNLP 2026 argues multiple-choice LLM benchmarks over-rely on plain accuracy while education research offers richer scoring schemes.
Testing schemes like penalty for wrong answers, retry-until-correct, and wagered points, the authors show these alternatives shift LLM rankings, align better with LLM Arena, and unmask model quirks such as overconfidence that accuracy hides.
More from Research
- Good Papers: a Rotten Tomatoes for ML papers with 20 AI reviewers — shangbinfeng · 2026-10-20
- Taco: hardware-agnostic LLM compression with one hash across 7 backends — phoenix10701 · 2026-10-20
- A multi-year deep dive into measuring AI slop across subreddits — Uhlo · 2026-10-20
- Scaling past 1T params: routing, hybrid attention, and 1M-token contexts — philipkiely · 2026-10-20
- An AI Cheated at StarCraft: Good Start Labs Caught GPT-6 Astra in Minutes by Watching It Play — danshipper · 2026-10-20
- Scripps Research gets 14M+ compute hours from Jensen Huang's foundation for AI biomedicine — EricTopol · 2026-10-20