FRI's Four-Year Study: Experts and Superforecasters Keep Underestimating AI Progress
Forecast Research Institute (FRI) released interim results from its four-year series on the accuracy of expert AI forecasts: domain experts and superforecasters systematically underestimated AI progress year after year, a conclusion repeatedly amplified by Ethan Mollick.
Confirmed
- FRI compiled forecasts collected in 2022 and revisited in mid-2025, covering economists, computer scientists, biologists and other experts; their predictions on the MATH, MMLU and QuALITY benchmarks consistently lagged actual progress
- Superforecasters underestimated progress even more than domain experts
- Asked in September 2025, top superforecasters gave only a 1.7% probability to AI solving a Millennium Prize Problem before September 2026; more optimistic industry experts gave just 4.6% — the same forecasters had previously badly underestimated AI lab capability gains
- Mollick also cited FRI's 2022 data: superforecasters and domain experts gave only about a 2.3% probability to AI winning an IMO gold medal by 2025; per related posts, IMO-level AI performance arrived 5-10 years earlier than forecast
Why it matters
- Persistent forecasting bias distorts policy, investment and risk assessment: conservative expert forecasts could leave AI governance and safety planning behind reality
- The study provides systematic empirical evidence for the debate over whether AI progress is over- or underestimated, pointing firmly to the latter
2026-09-24 ~ 2026-09-24 · 7 related posts
Primary sources
- Experts and superforecasters consistently underestimated AI progress by years — emollick ·
- Superforecasters Put AI Solving a Millennium Problem at 1.7% — and Keep Underestimating AI — emollick ·
- Forecasters gave AI IMO gold just 2.3% odds by 2025 — and short-term predictions are getting worse too — emollick ·
- Four-Year Study: Experts Including Superforecasters Repeatedly Underestimated AI Progress — Afinetheorem · 2026-09-24
- [source] Superforecasters Put AI Solving a Millennium Problem at 1.7% — and Keep Underestimating AI — emollick · 2026-09-24
- [source] Experts and superforecasters consistently underestimated AI progress by years — emollick · 2026-09-24
- Superforecasters Gave AI a 1.7% Chance of Solving a Millennium Problem — Badly Wrong — BorisMPower · 2026-09-24
- [source] Forecasters gave AI IMO gold just 2.3% odds by 2025 — and short-term predictions are getting worse too — emollick · 2026-09-24
2 near-duplicate retellings: michael_nielsen · austinc3301