Crucible Series 1: $25K Contest Reveals How to Make Top AI Forecasters Disagree
Crucible Series 1 was a $25,000 contest exploring where the strongest AI forecasting systems disagree about the future. Participants, including superforecasters and hedge fund quants, spent months hunting for 'wedge questions' that make AI forecasters diverge, and several effective strategies emerged. These findings also offer a methodology for building more discriminative AI forecasting evaluations.
Confirmed
- Background: many forecasting questions are effectively 'solved' (e.g., nuclear detonation probabilities) with models converging, while others are dominated by random uncertainty; purpose-built wedge questions are needed (per @tshevl).
- Strategy 1: exploit information asymmetry. When asked to forecast Hugging Face download counts, models with access to the latest APIs were more confident and accurate (per @tshevl).
- Strategy 2: test question decomposition. Models differ in whether they explicitly break questions into sub-forecasts; participants designed questions (e.g., 'when will a model surpass a certain benchmark') to amplify this difference (per @tshevl).
- Strategy 3: identifying discontinuities/jump events also effectively separates top forecasting systems (per @tshevl).
- Strategy 4: exploit bugs. Participants were adept at sniffing out edge cases where forecasters fail catastrophically due to technical issues, incentivizing developers to patch quickly (per @tshevl).
Why it matters
The contest exposed how existing forecasting benchmarks are too easy and cause AI predictions to converge, while discovered bugs directly drove improvements to forecasting systems.
2026-08-22 ~ 2026-08-22 · 6 related posts
Primary sources
- [source] A $25,000 contest reveals where top AI forecasting systems disagree about the future — tshevl · 2026-08-22
- $25k AI Forecasting Contest: How to Make Top Models Disagree — tshevl · 2026-08-22
- Tactics to Split AI Forecasters: Exploiting API Access — tshevl · 2026-08-22
- [source] Tactics to Split AI Forecasters: Information Asymmetry and Discontinuities — tshevl · 2026-08-22
- Tactics to Split AI Forecasters: Testing Decomposition — tshevl · 2026-08-22
- [source] Contest Winning Tactic: Exploiting AI Bugs and Edge Cases — tshevl · 2026-08-22