NeurIPS-rejected paper shows six agent dimensions to measure after benchmark saturation
random_walker · x · 2026-09-28
Arvind Narayanan's team says NeurIPS rejected their paper 'Life After Benchmark Saturation: A Case Study of CORE-Bench', which they argue remains an important contribution to evaluation science. The paper shows that retiring saturated benchmarks overlooks six key dimensions: construct validity (shortcuts), OOD generalizability, efficiency, reliability, model-vs-scaffold importance, and human-agent uplift. Using CORE-Bench Hard as a case study, they release CORE-Bench v1.1 and an OOD task suite, find the improved benchmark still measures efficiency and reliability post-saturation, and report statistically significant speedups from human-agent collaboration on real reproducibility tasks.
More from Research
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- Kind ICLR Reviewer Raises Score Despite Unresolved Concerns, Citing the Paper's Value — MarkSchmidtUBC · 2026-11-16
- Hand-editing 0.2% of weights plus 100 CPU-minutes turns an English TTS model into an 18MB Sanskrit one — kastnerkyle · 2026-11-16
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16