CruxBench: A Benchmark of Information Discovery
Hui Dai, Lina Piao, Nick Merrill, Nadja Flechner, Ezra Karger, Haifeng Xu
NeurIPS 2026
cs.CL, cs.AI
2026-09-27
CruxBench grades LLM subquestions by Polymarket KL updates at resolution. On 293 items, Opus-4.6 hits 0.016 bits vs 0.007 random timing and a 0.129 hindsight ceiling.
Most LLM benchmarks score a finished answer against a frozen label. Real decisions usually need an earlier move: pick the subquestions whose resolution would actually change a belief about a long-horizon target. The paper calls those subquestions cruxes.
Forecasting benchmarks already resist contamination with future events. They still grade whether the final probability is right. CruxBench grades information discovery. A model must propose a near-term, resolvable question that should move a prediction-market price when it resolves.
Ultimate questions are resolved binary Polymarket contracts from a 25 March 2026 snapshot. After dropping sports, crypto, and weather, then filtering on lifetime, liquidity, and event-level duplicates, 293 items remain. Each item gets one randomly sampled 30-day window.
Three stages.
Random timing is the mean hourly KL inside the window: how much the market moves if an hour is drawn at random. The hindsight oracle is the max hourly KL, a retrospective ceiling, not a reachable target.
Headline metric is KL, not entropy drop ΔH. KL is nonnegative on every sample and has lower sampling variance on small updates. In a 121-cell Monte Carlo, KL wins on variance in 78 cells.
Eight generators, six families, n=293.
| Model | VOI (bits) | Resolvable |
| Claude-Opus-4.6 | 0.016 | 95.2% |
| Gemini-3.1-Pro-Preview | 0.014 | 93.5% |
| GPT-5.4 | 0.013 | 91.5% |
| DeepSeek-V3.1 | 0.012 | 89.4% |
| Qwen3-235B-A22B | 0.012 | 92.5% |
| Llama-3.3-70B | 0.010 | 86.0% |
| Llama-3.1-8B-Instruct | 0.010 | 92.5% |
| GPT-3.5-Turbo | 0.009 | 93.9% |
| Random timing | 0.007 | n/a |
| Hindsight oracle | 0.129 | n/a |
Only the three frontier models beat random timing on an uncorrected one-sided paired t-test. Best score 0.016 bits versus a 0.129 oracle is close to an order of magnitude. Resolvability already sits between 86% and 95%. The score gap is whether the market moves when the crux resolves.
VOI tracks Epoch AI's composite capability index at r=0.90 (p<0.01), and tracks each generator's own start-of-window forecast of P(U) at r=0.76 (−Brier) and r=0.85 (AUC).
A downstream test gives each resolved crux to three smaller forecasters (Claude-Haiku-4.5, Gemini-3-Flash-Preview, GPT-5-Mini). Opus-4.6 cruxes cut mean Brier from 0.198 to 0.176 and lift AUC from 0.690 to 0.730. VOI correlates with those gains at r=0.65 and r=0.68. Every generator improves Brier over a no-crux baseline; AUC does not rise for every model. Many of GPT-5.4's high-KL cruxes rest on public statements of intent. Prices move; evidence about the ultimate outcome stays thin.
Mean pairwise cosine similarity of crux text is about 0.75. Semantic overlap is a weak proxy for the same question. Among pairs with cosine at least 0.8 that both name a numeric threshold, 84% pick different cutoffs and 24% receive opposite Yes/No labels. For the Louvre jewels market (any item returned by 31 December), asking whether police arrest a suspect catches the 25 October jump from about 0.24 to 0.43 (KL=0.135). DeepSeek-V3.1 requires arrest and charging; charges arrive four days later and the KL falls to 0.002.
Few evaluations are contamination-resistant by construction, open-ended in what a model may submit, and scored on quantified belief updates. For research assistants, intel decomposition, or forecasting agents, proposing the right near-term question is the first cut in the workflow.
Rank order follows general model strength. Absolute VOI sits next to random timing. Current models are not reliable crux engines. The contribution is an evaluation device: a quantity experts used to estimate by hand, VOI, turned into a scalable market statistic.
Population VOI is estimable. Per-crux VOI is not, because P(Q) and both conditionals are missing. Prices are a proxy; liquidity, simultaneous news, and window misalignment all leak into (p, p'). The study is retrospective, so live tests of the newest models remain future work.
Further discounts sit in the same appendices. 64.2% of cruxes resolve No because the named event never happens in-window; those non-events produce tiny updates and pick up background noise. Treating Q as a noisy signal of U, pooled Youden α̂ = TPR − FPR is only 0.030, with a 95% interval that includes zero. The three strongest models land at 0.11 to 0.12 and still fail to clear zero on their own. The downstream test feeds the future resolution to the forecaster on purpose, so it measures informational value, not a sequential gain available in real time. A shared news packet controls cutoffs; pretraining asymmetries remain. The resolver is Claude, and one generator is Claude; the paper does not ablate a cross-family resolver.