Jason Wei maps which benchmarks frontier labs actually report; four see universal adoption
davidmanheim · x · 2026-11-19
Jason Wei created a 'benchmark benchmark': a frequency distribution of which benchmarks frontier labs reported in 2026.
Key observations:
- HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption across frontier labs
- Scale AI is behind two of those four (SWE-Bench Pro, HLE with CAIS), plus MCP-Atlas (#9)
- OpenAI is the most prolific benchmark reporter among frontier labs
davidmanheim replied that domain-specific benchmarks like BioMysteryBench (bioinformatics) are never used by OpenAI and others, a shortcoming of providers unwilling or unable to test those capabilities.
More from Models
- One prompt: GPT-6.2 recreates Contra's 20-second intro purely in code — songguoxiansen · 2026-11-19
- Gabriel: benchmarks are broken — in the domains you know best, model quality is literally unmeasurable — gabriel1 · 2026-11-19
- Hinton: the 'just predicting next word' argument falls apart — LLMs can read a brand-new programming language — CatAstro_Piyush · 2026-11-19
- Jeremy Howard Confirms Gemini App's Neon Lives Up to the Hype — jeremyphoward · 2026-11-19
- Google ships a model every two weeks and nothing happens, argues dev — gabriel1 · 2026-11-19
- Miles Brundage complains Antigravity Enterprise accounts lack up-to-date models — Miles_Brundage · 2026-11-19