Measuring LLM Forecast Incoherence via Arbitrage Profits: New Paper from Sarkar & Andrews
soumitrashukla9 · x · 2026-09-05
Suproteem Sarkar announces a new paper with economist Isaiah Andrews evaluating probabilistic coherence in LLM forecasts. They build a forecasting environment from historical stock returns and quantify incoherence by the profit one can make arbitraging a model's forecasts — the larger the arbitrage, the more self-contradictory the predictions.
Related findings from the thread: forecasts grow more incoherent with more logical relations between events (joint vs. marginal distributions) and with irrelevant added context; coherence varies two orders of magnitude across models and higher coherence tracks higher accuracy.
Related event: Paper Tests LLM Probabilistic Coherence via Stock Arbitrage(2 posts)→
More from Research
- Self-explanation training generalizes beyond narrow hint formats to held-out evals — a_karvonen · 2026-09-05
- Two training targets from behavior investigations: counterfactual predictions and open-ended self-explanations — a_karvonen · 2026-09-05
- Anthropic Fellows train models to explain their own wild behaviors with generalization to held-out evals — a_karvonen · 2026-09-05
- Video DeltaNet open-sources hybrid-attention VDN-H3, 14.4s video in 11.2s on 8 B200s — BigWideBaker · 2026-09-05
- Deep Learning Weekly #471: Claude Fable 5.1 launch, production-parity LLM evals, alignment paper — dl_weekly · 2026-09-05
- CoRL 2026 paper decisions out on OpenReview; conference heads to Austin this November — yukez · 2026-09-05