OpenAI's 96 ECI benchmarks skew to math and AI research; hard multimodal and cybersecurity tests are rare
scaling01 · x · 2026-11-19
OpenAI data shows model time horizons doubling every 3.8 months, and the 96 benchmarks behind its ECI reveal its priorities: math and AI research first, then science, professional work and engineering.
Benchmark counts by category (GPT-6.2 Astra at 170.3 as reference): Science 18, software/hardware engineering 16, professional work 10, language & reasoning 9, multimodal 9, math 8, AI research 8, computer/tool use 7, games 7, cybersecurity only 4. The poster is surprised how few hard multimodal, game and cybersecurity benchmarks exist.
More from Models
- Gemini release gaps halve each cycle: 28 to 15 to 6 days per new model — davidyin44 · 2026-11-19
- Surge AI benchmarks Gemini 4 Neon: tops Tuesday@Work at 71.6, trails on SWE — echen · 2026-11-19
- Google's rumored Neon and Carbon models reportedly match or beat Opus in early tests — almmaasoglu · 2026-11-19
- Musk says Grok 4.8 is in RL phase, built for 'usefulness max' over benchmarks — XFreeze · 2026-11-19
- Liquid AI's d1.2-ultra beats Google and Microsoft decision models at half the price — JosephJacks_ · 2026-11-19
- Prediction: Anthropic to drop Opus 5.6 tomorrow — the mysterious 'Model 4' from METR's RSI report — daniel_mac8 · 2026-11-19