Eval Cards tracks 309K model-benchmark results; 78% of evals can't be reproduced

davidmanheim · x · 2026-11-19

In a thread responding to Jason Wei's 'benchmark benchmark' (a frequency distribution of benchmarks reported by frontier labs in 2026), davidmanheim highlights Eval Cards, a reporting layer over AI evaluation infrastructure and a NeurIPS spotlight paper.

Key stats (corpus snapshot Nov 18, 2026):

The thread also notes HLE, SWE-Bench Pro, DeepSWE and Terminal-Bench saw universal frontier-lab adoption, two of them backed by Scale AI, and that OpenAI is the most prolific benchmark reporter while bioinformatics benchmarks like BioMysteryBench go untested.

Original post →

More from Research

Research channel →