NeurIPS position paper: AI evals should mandate item-level data releases, built OpenEval with 10.9M responses
sanmikoyejo · x · 2026-10-23
Researchers from JHU, UIUC, Microsoft Research Asia, Tsinghua, and Stanford argue in a NeurIPS 2026 position paper that AI evaluation should make standardized item-level benchmark data the default infrastructure instead of relying on aggregate scores.
Key arguments
- Current evaluations suffer from underspecified item selection, construct misalignment, and poor generalization; aggregate-only reporting keeps these failures undetected, inflating capability claims and misdirecting research.
- Item-level data (item content, test conditions, model responses, per-item statistics) enables validation, auditing, and replication that aggregate scores cannot.
- Transparency and responsible sharing aren't mutually exclusive: controlled access can mitigate benchmark contamination while preserving item-level evidence.
OpenEval
To demonstrate feasibility, the team built OpenEval, an open-source item-level repository hosting 160K+ items and 10.9M+ model responses under a unified JSON schema, with faithful, timestamped snapshots of full experimental context — a shared foundation the community can build on for comparable, reusable evaluation research.
More from Research
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23
- Perceptic launches PercepticOS, an AI workspace for 'scientific superintelligence' — nathanbenaich · 2026-10-23
- EPFL's Bosselut to talk long context as test-time learning at ELLIS seminar — andre_t_martins · 2026-10-23
- DRATS: Adaptive Task Sampling Boosts Worst-Task Performance in Multi-Task RL — Cohere · 2026-10-23