Jason Wei maps which benchmarks frontier labs actually report; four see universal adoption

davidmanheim · x · 2026-11-19

Jason Wei created a 'benchmark benchmark': a frequency distribution of which benchmarks frontier labs reported in 2026.

Key observations:

davidmanheim replied that domain-specific benchmarks like BioMysteryBench (bioinformatics) are never used by OpenAI and others, a shortcoming of providers unwilling or unable to test those capabilities.

Original post →

More from Models

Models channel →