AI Benchmarks Face Saturation and Soaring Evaluation Costs

An ICML study finds nearly half of LLM benchmarks are saturated, with longevity determined by expert curation rather than public data, while researchers clarify saturation is distinct from data contamination; evaluation costs, such as $2,829 per GAIA run, are becoming a new compute bottleneck.

2026-10-19 ~ 2026-10-19 · 3 related posts

• ICML 2026 study: nearly half of LLM benchmarks are saturated — expert curation, not public data, decides longevity — evijit · 2026-10-19
• Researcher: Openness uncorrelated with benchmark saturation, distinct from contamination — evijit · 2026-10-19
• AI evals are the new compute bottleneck: GAIA run costs $2,829, HAL spent $40K — evijit · 2026-10-19