Eval Cards tracks 309K model-benchmark results; 78% of evals can't be reproduced
davidmanheim · x · 2026-11-19
In a thread responding to Jason Wei's 'benchmark benchmark' (a frequency distribution of benchmarks reported by frontier labs in 2026), davidmanheim highlights Eval Cards, a reporting layer over AI evaluation infrastructure and a NeurIPS spotlight paper.
Key stats (corpus snapshot Nov 18, 2026):
- 9,946 models, 309,050 (model, benchmark, metric) triples, 52 reporting organizations, 2,116 model developers, 93 benchmark families
- Only 22% of results disclose every setup item; 78% leave at least one undisclosed — most often output length (74%) and decoding settings (73%)
- Completeness averages 41% (range 7–93%); just 10% of triples have multi-party reports; 17% of setup-eligible groups diverge across variants
The thread also notes HLE, SWE-Bench Pro, DeepSWE and Terminal-Bench saw universal frontier-lab adoption, two of them backed by Scale AI, and that OpenAI is the most prolific benchmark reporter while bioinformatics benchmarks like BioMysteryBench go untested.
More from Research
- François Fleuret: perfect inductive bias causes brutal loss drops that gradient descent reads as spikes — francoisfleuret · 2026-11-19
- WebGPU playground lets you experiment with weight-space programming on Qwen3.5-0.8B — wsxiaoys · 2026-11-19
- ViT can express any conv layer mathematically, but CNN priors still win on small edge devices — burny_tech · 2026-11-19
- Jason Wei maps which benchmarks frontier labs actually report; four see universal adoption — davidmanheim · 2026-11-19
- PIPETTE fixes RL red-teaming mode collapse to harden LLM agents against prompt injection — Lipeng He · 2026-11-19
- Movie2Game-Bench tests 14 models on film-to-game adaptation; playable sequences score just 37.9/100 — Jialing Xie · 2026-11-19