Can LLMs Anticipate Their Cruxes?
David Reber, James Reber, Ari Holtzman, Victor Veitch
cs.CL, cs.LG
2026-10-15
Across 59 games, cruxes named in advance overlap real forecast disagreements at 13.4%, below a 39.2% shuffle. One rollout is closer and still misses most of the set.
A model's written case for a decision is often too long to use. A coding agent that skips inputs crashing its pipeline has decided those inputs do not matter, and it rarely states that decision as a question. This paper, from the University of Chicago, defines that kind of sub-question as a crux: a yes/no factual question on which independent samples (rollouts) take opposite sides, and where flipping the answer might have changed the conclusion. A crux is a property of the distribution of rollouts. One chain of thought is not enough, by definition, to see it.
Two cheap procedures are the obvious ones. Ask in advance where the decision will turn, or run one rollout and pull the premises it used. The test of either is the disagreements the model's own samples actually produce.
The task is binary forecasting, finished before the events resolved, so the source of disagreement between rollouts is identifiable. Fifteen Metaculus questions get about 30 rollouts each, written by gpt-5-nano with separate web searches, so variation comes from sampling and from retrieved pages. Fifty-nine college-basketball games get nine rollouts each. Three scouting reports per team (a traditional analyst, a quantitative forecaster, a matchup analyst) were cached before the tournament, and every rollout reads those reports. On basketball, gpt-5.4-nano forecasts, names cruxes in advance, and extracts them, and the advance prompt sees every report. On Metaculus, gpt-5.4 names cruxes in advance and gpt-5.4-mini extracts them; the advance prompt never sees the search results.
Two cruxes match when the same evidence would confirm or falsify both. A generative judge cannot score every pair: about 12,978 judgments per basketball game and about 1.01 million per Metaculus event. Qwen3-Reranker-8B scores a pair in one forward pass of about 270 tokens, averaged over both directions. Capping each set at 25 cruxes brings the main evaluation to about 6.4 GPU-hours, an estimated 1/86 of the compute of a local judge.
Average-linkage clustering cuts the tree without seeing which set a crux came from. Headline numbers use the middle bar: resolving one crux would necessarily resolve the other. A stricter bar asks for the same point of disagreement; a looser one asks only that both would end the same dispute. Many cruxes sit in clusters of one, and a singleton can never be shared, so raw overlap is partly a fact about cluster size. The permutation baseline freezes the clusters and reassigns the two labels at random. The gap is baseline minus observed overlap. Below the baseline means the two sets land together less often than chance.
On basketball, at the middle bar with 25 cruxes a side, advance cruxes overlap discovered ones 13.4%±1.6%, against a 39.2% baseline, a gap of 25.9±1.2 points (p<10^-10). Fifty-five of 59 games fall below their own baseline, three tie it, one sits above. The median game shares 10.5% of its clusters, and nine share none. Growing each set from 5 cruxes to 25 moves overlap from 6.3% to 13.4% and the baseline from 13.5% to 39.2%. More names do not close the gap.
| Source | Match rule | Overlap | Shuffle baseline | Gap (points) |
| Basketball, 59 games | same question | 5.6% | 33.4% | 27.8 |
| Basketball, 59 games | same resolution | 13.4% | 39.2% | 25.9 |
| Basketball, 59 games | same dispute | 12.8% | 41.2% | 28.4 |
| Metaculus, 15 events | same question | 21.9% | 58.0% | 36.1 |
| Metaculus, 15 events | same resolution | 39.9% | 70.9% | 31.0 |
| Metaculus, 15 events | same dispute | 39.7% | 68.9% | 29.3 |
Metaculus gaps by 31.0±4.9 points, with 13 of 15 events below their own baseline. Neither a stricter nor a looser match closes it.
A single rollout leaks into the pool, because some many-rollout cruxes were extracted from pairs that include it. After those are removed, basketball overlap is 18.8% against 25.1%, a gap of 6.4 points, still low in 47 of 59 games. Coverage of the many-rollout cruxes is 28.6% against 37.8%. On that same stripped pool the advance set gaps by 26.8 points. On Metaculus the single rollout sits at 48.5% against 44.0%, on the baseline: a random draw from the pool, not the whole pool.
Halves of the many-rollout set overlap 3.4 times as much as the two sets do with each other; halves of the advance set overlap 4.4 times as much. Each set is internally coherent, and the two sets miss each other. For Iowa State against Kentucky, the advance crux asks whether free throws become decisive; one discovered crux is that Iowa State forced 23 turnovers in an earlier game. Forcing the advance prompt to be as specific as the discovered cruxes (96.6% then state a number, against about 32% before) leaves overlap near 17%. The gap falls from 24.7 on a same-day control to 16.4, because the baseline drops, and it remains in 58 of 59 games. Matching on the evidence behind a crux, rather than its wording, also leaves a gap. Season statistics, ranks, or records show up in 75% of discovered evidence statements and 11% of advance ones.
The 29 games whose rollouts do not all pick the same winner overlap 16.0%±2.5% against 40.5%. The gap's direction holds for qwen3:32b, gemma4:31b, and gpt-oss:120b. All 27 cells of three predictors by three extractors by three match rules fall below their baselines. Swapping the extractor from gpt-5.4-nano to qwen3:32b moves the gap at 25 cruxes by about 11 points. Distinct cruxes grow about linearly with rollouts: basketball reaches about 28 clusters at nine rollouts and has not saturated. Five of those nine cover 61% of the distinct cruxes; seven cover 83%. Metaculus, scored on twelve rollouts, settles near five.
Questions named in advance often sound sensible. They are still the wrong questions for where that model's samples actually split. In a high-stakes decision, asking only those questions audits a self-description. One chain of thought cuts the basketball gap to about six points. That is enough to inspect a single argument. It is not a full list of the cruxes.
The definition requires the distribution. Five to seven rollouts already cover most distinct cruxes, and a reranker can decide which wordings are the same crux. The direction of the result, overlap below a label shuffle, holds across both sources, three match rules, and four model families.
Every crux counts once. A premise that would move a probability from 70% to 30% weighs the same as one that would barely move it. Whether the gap survives weighting by value of information is untested.
On Metaculus the advance prompt never saw the pages the rollouts retrieved, so some of the miss is an information gap. Basketball locks the reports and shows them to the predictor, and the gap remains, so the information gap is not the whole story. The two sources should not be compared on the size of the gap alone. A second draw of the Metaculus advance cruxes moves that gap by about ten points. The direction is stable from 5 to 25 cruxes and across match rules; with 15 events the size is not.
No person labeled equivalence. The three instructions were picked from ten candidates by agreement with gpt-5.4-nano acting as a judge, measured on 2,000 basketball pairs. Of the 34 pairs the judge called equivalent, only three cross the two sets, so accuracy on the pairs that decide overlap was not measured. The reranker keeps apart about 95% to 98% of pairs the judge called distinct, and recalls only about 0.68 to 0.74 of the pairs it called equivalent.
Every request is a forecast of one yes/no event. Reports, plans, and code changes have no shared number for two rollouts to disagree on, and the paper leaves open whether pairwise contrast extends there. There is also no user study of whether showing someone the discovered cruxes improves a decision. Basketball is in the study because the evidence can be held fixed.