$25k AI Forecasting Contest: How to Make Top Models Disagree
tshevl · x · 2026-08-22
A $25,000 contest was held to discover where the best AI forecasting systems disagree. It attracted superforecasters and hedge fund quants probing for "wedge questions".
Motivation:
- Many questions are "solved" (e.g., nuclear explosion probability), leading to similar predictions.
- Some questions are intractable due to aleatoric uncertainty (e.g., stock prices), where models just agree on being uncertain.
Winning Tactics:
- Exploit information asymmetries: Forecasting Hugging Face downloads favored models with API access.
- Identify discontinuities: Predicting outcomes during rule changes (e.g., NIST classifications).
- Test decomposition: Asking about complex events (e.g., beating a leaderboard) forces problem breakdown.
More from AGI Musings
- Dev: live voice + tool-calling + image gen now combine into a workable 'Her'-like experience — andreisavu · 2026-10-16
- Minerva brings AI to a 3,000-person village and acquires accounting firms across the US — ycombinator · 2026-10-16
- Interpretability is too costly for everything, but great for post-hoc forensics — jessi_cata · 2026-10-16
- Neuron labels are like code comments: debate on interpretability vs CoT — jessi_cata · 2026-10-16
- Jensen Huang: UK Must Fix Energy Prices to Seize Its AI 'Goldilocks Moment' — KeeganMcB · 2026-10-16
- SF Vibes Are Electric as Engineers Move From Automation Anxiety to Awe — zack_overflow · 2026-10-16