ICML 2026 study: nearly half of LLM benchmarks are saturated — expert curation, not public data, decides longevity
evijit · x · 2026-10-19
A discussion thread centered on the ICML 2026 paper When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation:
- The study formally defines benchmark saturation and analyzes 60 LLM benchmarks across 14 saturation-related properties, finding nearly half exhibit saturation, with rates increasing as benchmarks age.
- Key finding: resilience to saturation is driven by expert curation, not by public test data — openness is uncorrelated with saturation.
- Design choices can extend benchmark longevity, informing more durable evaluation approaches.
The surrounding thread also covers: open evals amount to "trusting every lab not to cheat," so even low-quality hidden/uncheatable benches may be more useful relative datapoints; agentic evals require their own compute to run complex harnesses; third-party auditor selection carries conflict-of-interest risk, and the author would trust a proven contamination detection method over an honor system; the DeepMind/OpenMined AVERI double-blind evals project is tackling similar questions.
Related event: AI Benchmarks Face Saturation and Soaring Evaluation Costs(3 posts)→
More from Research
- Oxford DPhil thesis argues Attribution-Based Control could fix hallucination, alignment and power concentration at once — iamtrask · 2026-10-19
- Stanford's Memorilla: 16 memory tokens beat full-context with 108x fewer tokens — omarsar0 · 2026-10-19
- Philosopher's "AI and Consciousness: A Skeptical Overview" Free Online This Week — eschwitz · 2026-10-19
- Handpicked 40-Game Puzzlescript Benchmark Aims to Be "ARC-AGI-3.5" for Agent Reasoning — scaling01 · 2026-10-19
- LeCun: distributed representations beat grandmother cells, known since the 1980s — burny_tech · 2026-10-19
- Researcher Aran Nayebi: consciousness paper shows sustained dominance, not 'flickering' — aran_nayebi · 2026-10-19