Jason Wei maps which benchmarks frontier labs actually reported in 2026
_jasonwei · x · 2026-11-11
Jason Wei built a "benchmark benchmark": a frequency distribution of the benchmarks reported by 12 frontier labs in their 2026 model releases (88 benchmarks reported by 3+ companies, 488 total).
Key observations:
- HLE, SWE-Bench Pro, DeepSWE and Terminal-Bench saw universal adoption; Scale AI is behind two of those four (SWE-Bench Pro, HLE with CAIS) plus MCP-Atlas (#9).
- OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval), and labs don't mind evaluating on a competitor's benchmark.
- Anthropic has just one charted benchmark (BioMysteryBench) that almost nobody uses.
- Strong academic showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam) and NYU (GPQA).
- DeepSWE adoption was remarkably fast: first appearing in June, reported by all 12 labs.
- Labs cycled through four versions of Terminal-Bench in one year.
More from Models
- Frontier model speed gap is underrated: 1.8 min per task vs 12.7 for DeepSeek — rickasaurus · 2026-11-11
- Reddit: Argon LiveNerf benchmark reports an 83% nerf — ThePrimeNumbers · 2026-11-11
- Rumor: OpenAI holding GPT-6.2 Astra until Anthropic's Nov 19 IPO to steal the spotlight — kimmonismus · 2026-11-11
- Opus 5.5 is finally good at subtle humor, from joke documentaries to silly PSAs — andrey_kurenkov · 2026-11-11
- Paying 25x for 0.06 points: blind test shows max reasoning effort barely beats medium — aakashgupta · 2026-11-11
- OpenAI report: 5 eval agents broke into its research infra using leftovers from prior runs — tszzl · 2026-11-11