When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, Vilém Zouhar, Srishti Yadav, Chenxi Whitehouse, Dayeon Ki, Jennifer Mickel, Leshem Choshen, Marek Šuppa, Jan Batzner, Jenny Chim, Jeba Sania, Yanan Long, Hossein A. Rahmani, Christina Knight, Yiyang Nan, Jyoutir Raj, Yu Fan, Shubham Singh, Subramanyam Sahoo, Eliya Habba, Usman Gohar, Siddhesh Pawar, Robert Scholz, Arjun Subramonian, Jingwei Ni, Mykel Kochenderfer, Sanmi Koyejo, Mrinmaya Sachan, Stella Biderman, Zeerak Talat, Avijit Ghosh, Irene Solaiman
ICML 2026
cs.AI
2026-02-19
An ICML study builds an uncertainty-aware saturation index for 60 LLM benchmarks: nearly half can no longer separate top models, hiding test data does not help, and expert curation ages best.
Benchmarks like MMLU and HumanEval are the yardsticks for measuring how far language models have come. But the top of the leaderboard keeps compressing. When the strongest models all land within a point or two of each other, a benchmark stops telling you who is actually better. Researchers call this saturation, and it has never had a settled operational definition.
Does saturation mean models have genuinely mastered the task and hit a ceiling, or does it just mean several models are statistically tied? Why do some benchmarks blur together within a year while others keep separating models for years? The community has plenty of folk theories: public test sets get memorized, multiple choice is too easy to max out, popular benchmarks die first. Almost none of it had been tested systematically. This ICML 2026 paper (38 authors across ETH, Stanford, Hugging Face and others) tries to turn those vague intuitions into something measurable.
The authors first separate saturation from stagnation. Saturation is when top models approach an empirical ceiling and become indistinguishable; stagnation is when they are statistically tied but not yet near the ceiling. The distinction matters: the first can mean a task is genuinely solved, the second usually means the measurement is too coarse.
They define an uncertainty-aware saturation index Sindex in [0,1], computed as exp(−Rnorm²). Rnorm is the score gap between the top model and the k-th model (default k=5) divided by the standard error of that gap. When the gap is smaller than the evaluation noise, the index approaches 1, meaning the top of the field is bunched together. The standard error uses an effective test set size neff = n·α to downweight enormous benchmarks, so a 50,000-item set does not crush its noise toward zero and look artificially unsaturated. Benchmarks fall into five bins from very low (<0.01) up to very high (≥0.9).
The sample is 60 text-based LLM benchmarks assembled two ways: benchmarks appearing in technical reports from OpenAI, Anthropic, Google, Meta and Alibaba between January 2022 and November 2025 (61 documents screened, 190 benchmarks found), plus highly cited benchmark papers from Semantic Scholar. The final 60 comprise 56 public and 4 private, 44 English-only and 16 multilingual, 28 closed-ended and 31 open-ended, 14 templated, ranging from 1 to 114 months old. Around 14 properties (accessibility, language scope, age and popularity, task design, data construction and quality) feed six hypotheses.
Of the 60 benchmarks, 29 show high or very high saturation (Sindex ≥ 0.7), and 14 are very high (≥ 0.9). Nearly half.
The commonly believed safeguards do not hold up:
| Hypothesis | Result |
| Public saturates faster than private (hiding the test set prevents memorization) | No meaningful difference between public (56) and private (4); secrecy does not help |
| English-only saturates faster than multilingual | Multilingual benchmarks are just younger (32.9 vs 48.9 months); controlling for age removes the gap |
| Closed-ended saturates faster than open-ended | No significant difference (age-balanced, p=0.40) |
| Templated benchmarks saturate faster | No significant difference (p=0.10) |
| More popular (cited, in reports) saturates faster | No association once age is controlled (ρ=0.22, p=0.12) |
| Expert or human-curated resists better than crowdsourced or synthetic | At comparable ages, expert-curated benchmarks saturate less; ARC-AGI and BIG-Bench Hard stay unsaturated despite long exposure |
The two most consistent predictors are benchmark age and test set size: older means more saturated, larger test sets mean less. The saturated share rises from 42.9% for benchmarks under 24 months old to 54.5% for those over 60 months, with mean Sindex climbing 0.51, 0.52, 0.60 across the bins (directionally consistent but not significant at conventional thresholds). A Bayesian regression over all properties reaches R²Bayes of 0.884 ± 0.012, with age and test set size carrying the most weight.
Benchmarks with documented quality issues (40) saturate more than those without (20), but they are also older (51.5 vs 30.9 months, p=0.01). Whether poor quality causes saturation, or old benchmarks simply get both overfit and audited, is not something the data can separate.
The valuable takeaway is a debunking: hiding the test set, switching question format, or adding template variety will not extend a benchmark's life. Two things actually delay saturation, keeping benchmarks expert-curated and adversarially refreshed, and making the test set larger and more granular. For anyone building evaluations or choosing models, a 0.5-point gain on some benchmark is not automatically progress; if it sits inside the evaluation noise, it is nothing. The authors argue leaderboards should report confidence intervals and the spread of top scores by default, and that gains smaller than the noise should not count as a breakthrough.
The paper also offers a rule for when to retire a benchmark. Saturation is not inherently bad: if the benchmark measures a clearly defined capability and models reach its ceiling, that can mean the task is genuinely mastered. What is bad is when measurement resolution collapses first, the set is too small or too easy, the top models cluster inside the noise, and everyone concludes they are equally strong. Telling those two cases apart is what the paper really wants to push.
The authors list several. The benchmark sample reflects current evaluation habits and may overrepresent widely adopted benchmarks. Saturation is computed from leaderboard snapshots, so benchmarks updated irregularly can hide their dynamics. The index depends on publicly available frontier-model evaluations, which may be incomplete or selectively reported. They also assume benchmark properties are fixed in time, though attributes like annotation diversity keep evolving after release.
A few more concerns stand out. Private benchmarks number only 4, so the "no difference versus 56 public ones" conclusion rests on a badly unbalanced comparison. All six hypotheses are observational correlations with no causal identification, and "expert curation resists saturation" could simply mean those benchmarks also get refreshed more often. The uncertainty estimates only cover accuracy-like metrics, so Elo, pass@k, and judge-based scores all need their own variance work that the current index does not provide.