Measuring LLM Forecast Incoherence via Arbitrage Profits: New Paper from Sarkar & Andrews
soumitrashukla9 · x · 2026-09-05
Suproteem Sarkar announces a new paper with economist Isaiah Andrews evaluating probabilistic coherence in LLM forecasts. They build a forecasting environment from historical stock returns and quantify incoherence by the profit one can make arbitraging a model's forecasts — the larger the arbitrage, the more self-contradictory the predictions.
Related findings from the thread: forecasts grow more incoherent with more logical relations between events (joint vs. marginal distributions) and with irrelevant added context; coherence varies two orders of magnitude across models and higher coherence tracks higher accuracy.
Related event: Paper Tests LLM Probabilistic Coherence via Stock Arbitrage(2 posts)→
More from Research
- Richard Socher: AI could compress a century of scientific breakthroughs into a decade — RichardSocher · 2026-10-21
- Startup Electric Plant builds an AI model to decipher plants' electrical signals — TechCrunch AI · 2026-10-21
- Claude reads the literature to build open human surfaceome atlas, claim by claim — owl_posting · 2026-10-20
- Eval researcher teases guide to subjective-task evals, with a curated resource list — cwolferesearch · 2026-10-20
- IIT Madras article splits LLM memory into five approaches and asks why belief revision is still missing — LimeOptimal539 · 2026-10-20
- Scholar writes AI-assisted book on storytelling values, work nominated for EMNLP best paper — _akpiper · 2026-10-20