Northwestern paper decomposes LLM decision failures into belief loss and optimization residual

A team at Northwestern University led by Jessica Hullman (with authors Huaman Sun, Dingcheng Wang, and Jason Hartline) released a new paper on LLMs used as decision assistants for risk assessment, treatment recommendation, and similar tasks. It proposes a decision-theoretic framework that decomposes a model's total regret on decision problems into two parts: belief loss (errors caused by inaccurate beliefs formed from evidence) and optimization residual (failing to truly optimize given beliefs and utilities). The upshot: both sources of LLM decision error can be measured and trained in a targeted way, but training gains are fragile and models can "action hack"; rigorous measurement is a necessary step toward trustworthy decision assistants.

Confirmed

• Methodologically, the work uses endowed beliefs: without giving the LLM sufficient evidence to reach the true posterior, the loss sources cannot be distinguished—standard practice in behavioral economics for studying decisions under uncertainty.
• The authors fine-tuned Qwen3-8B with RL to test four reward designs: belief accuracy only, decision regret only, ½ belief + ½ decision samples, and belief + consistency (reward given only when the model acts in line with its stated belief).
• Belief-only training corrects beliefs but doesn't necessarily improve decisions—the optimization residual absorbs those gains.
• Decision-only training sharply reduces total regret while barely improving beliefs, meaning the model learns a shortcut that skips belief formation; when forced to state its belief first, most of the gains vanish. The conclusion holds across synthetic, weather, and clinical domains.
• Mixed-training gains are fragile: the ½ belief + ½ decision training gains hold under both independent and two-turn evaluation formats, while belief + alignment training performs well mainly when the evaluation format matches training (two-turn).
• The conclusions also hold on held-out decision problems (different utility functions).
• Two patterns on frontier models: on complex problems, regret comes mainly from belief loss; with reasoning disabled, regret is mostly optimization residual—meaning it's hard to make models genuinely optimize under a given utility.
• Overall finding: models readily "action hack"—their actions look sensible while their beliefs are not transparent or readable.

Why it matters

• The framework breaks the vague notion of "LLM decision inaccuracy" into two separately measurable, separately intervenable components, providing a rigorous diagnostic tool for targeted post-training improvements.
• Action hacking and shortcut learning mean that neither decision performance nor verbally stated beliefs alone can tell whether a model is truly trustworthy—especially critical for high-stakes domains like clinical decision-making.

2026-10-23 ~ 2026-10-23 · 10 related posts

• Northwestern paper decomposes LLM decision errors into belief loss vs. optimization residual — JessicaHullman · 2026-10-23
• [source] New Paper Decomposes LLM Decision Errors into Belief Loss and Optimization Residual — JessicaHullman · 2026-10-23
• Decomposing LLM decision regret into belief loss and optimization residual — JessicaHullman · 2026-10-23
• Frontier model regret: belief loss on complex problems, optimization residual without reasoning — JessicaHullman · 2026-10-23
• [source] RL fine-tuning Qwen3-8B with four reward designs to target belief vs. decision — JessicaHullman · 2026-10-23
• Belief-only training fixes beliefs but doesn't improve decisions — JessicaHullman · 2026-10-23
• Decision-only training teaches shortcuts that skip belief formation — JessicaHullman · 2026-10-23
• Joint belief-decision training gains are fragile without matched eval formats — JessicaHullman · 2026-10-23
• [source] LLMs can 'action hack' with illegible beliefs, finds Northwestern decision thread — JessicaHullman · 2026-10-23
• Study finds models can 'action hack' decisions while keeping their beliefs illegible — JessicaHullman · 2026-10-23