Stanford's 457-page AI Index: cost down ~30%/yr, top-model gap narrows to 5.4%

mdancho84 · x · 2026-10-27

mdancho84 breaks down the key takeaways from Stanford HAI's 457-page AI Index Report 2025:

Cost & efficiency: price-performance improves 30%/year and energy efficiency 40%/year — AI is moving from "demo" to "default."
Compute still exploding: training compute for notable models doubles roughly every 5 months; only a few players can afford frontier scaling.
Industry dominates: 90% of notable AI models in 2024 came from industry (vs 60% in 2023).
Models converging: the Chatbot Arena gap between #1 and #10 narrowed from 11.9% to 5.4% in a year — workflow, evals and data matter more than model choice.
Open vs closed: the gap between leading open-weight and closed-weight models narrowed to 1.70% (Feb 2025), changing build strategies and budgets.
Benchmarks crushed: hard benchmarks like MMMU/GPQA/SWE-bench jumped +19 to +67 points from 2023 to 2024, making public benchmarks unreliable.
Career playbook: for 2026 data scientists, the leverage skills are building your own eval harness, workflow engineering with agents/pipelines, and inference cost awareness.

More from AGI Musings