Stanford's 457-page AI Index: inference cost-performance up ~30%/year, open-vs-closed gap narrows to 1.7%
mdancho84 · x · 2026-09-03
Matt Dancho breaks down key findings from Stanford's 457-page AI Index report and what they mean for data science careers in 2026.
- Cost & efficiency: price-performance improves 30%/year and energy efficiency 40%/year — AI is moving from demo to default.
- Compute still exploding: frontier training compute doubles every 5 months; only a few players can afford it.
- Industry dominance: 90% of notable models came from industry in 2024 (vs 60% in 2023).
- Frontier convergence: the gap between #1 and #10 on Chatbot Arena narrowed from 11.9% to 5.4% in a year — workflow, evals and data matter more than model choice.
- Open vs closed: gap narrowed to 1.70% on Chatbot Arena (Feb 2025).
- Benchmarks getting crushed: +19 to +67 points on MMMU/GPQA/SWE-bench from 2023 to 2024.
- Career playbook: build your own eval harness, invest in workflow engineering (agents + pipelines), and master inference economics.
More from AGI Musings
- AI supercycle analysis: application diffusion, not infrastructure, is the near-term bottleneck — menhguin · 2026-09-03
- AI has cut entry-level job postings by up to 9%, Forbes calls it the 'invisible layoff' — KeanuRave100 · 2026-09-03
- Researcher: An LLM saying 'I'm hungry' doesn't mean it's actually hungry — ValerioCapraro · 2026-09-03
- Viral poem 'Models Learn What They Live' distills the alignment problem — PeterBowdenLive · 2026-09-03
- Cohere Labs head: multilingual AI's 'longitude problem' — scale isn't enough — Cohere_Labs · 2026-09-03
- If Users Only Reach Your Product via Agents, Are They Still DAUs? Belsky Says Yes — _AustinCalvert_ · 2026-09-03