$25K Crucible Series 1 Contest Reveals How to Make Top AI Forecasters Disagree

Crucible Series 1 was a forecasting competition with a $25,000 prize pool, run to explore where the strongest AI forecasting systems disagree about the future. Participants included superforecasters and hedge fund quants who spent months hunting for "wedge questions" capable of driving AI forecasters apart, and several effective strategies emerged.

Confirmed

• Background: many forecasting questions are already "solved" (e.g. the probability of a nuclear detonation), so models converge on their predictions; other questions are too noise-dominated to separate model skill, which is why purpose-built wedge questions are needed (per @tshevl).
• Strategy 1: exploit information asymmetry. For example, when forecasting Hugging Face download counts, models with access to up-to-date APIs are more confident and more accurate (per @tshevl).
• Strategy 2: test question decomposition. AI models handle sub-forecasts differently—some explicitly break questions into sub-predictions and some don't—so participants designed questions (e.g. "when will a model surpass a given benchmark") to amplify those differences (per @tshevl).
• Strategy 3: identifying abrupt changes or discontinuous events also effectively separates top forecasting systems (per @tshevl).
• Strategy 4: exploit bugs. Participants were remarkably good at sniffing out edge cases where AI forecasters fail catastrophically due to technical issues, and those findings pushed developers to patch things quickly (per @tshevl).

Why it matters

The competition exposed how existing forecasting benchmarks are too easy, causing AI forecasts to converge, and offers a methodology for building more discriminative AI forecasting evaluations—while the system vulnerabilities it surfaced directly drove improvements to forecasting systems.

2026-08-22 ~ 2026-08-22 · 6 related posts

• [source] A $25,000 contest reveals where top AI forecasting systems disagree about the future — tshevl · 2026-08-22
• $25k AI Forecasting Contest: How to Make Top Models Disagree — tshevl · 2026-08-22
• [source] Tactics to Split AI Forecasters: Exploiting API Access — tshevl · 2026-08-22
• Tactics to Split AI Forecasters: Information Asymmetry and Discontinuities — tshevl · 2026-08-22
• Tactics to Split AI Forecasters: Testing Decomposition — tshevl · 2026-08-22
• [source] Contest Winning Tactic: Exploiting AI Bugs and Edge Cases — tshevl · 2026-08-22