Hybrid Decision Model learns when to reason: 72.5% lower latency at higher accuracy

hbouammar · x · 2026-11-13

Researchers present "The Value of Thinking" (arXiv 2611.16961), introducing a zero-or-T thinking paradigm: a single frozen backbone either answers in one forward pass (0 tokens) or runs full autoregressive reasoning only when the estimated value of thinking exceeds a per-request price λ.

Key design

A Hybrid Decision Model (HDM) uses a lightweight self-routing block that estimates the expected utility of slow-mode reasoning from implicit thinking, trained via Calibrated Decision Policy Optimisation (CDPO) — an RL recipe using episode-level returns for agentic tasks lacking per-decision labels.

Results

JevBench: HDM-4B scores 0.857 vs Jev's 0.851, with median latency of 0.179s vs 0.652s (-72.5%).
Reaching 90% of always-thinking accuracy needs only 103 thinking tokens vs 591 — a 5.7x reduction.
On agentic benchmarks, token cost drops 80.1% on MiniWoB and 52% on ALFWorld; HDM-27B routed hits 0.987 @ 2.49s.
88% of the gain comes from just 62% of tokens across MMLU-Pro, AGIEval, and GPQA-Diamond with a single price sweep.

Original post →

Hybrid Decision Model learns when to reason: 72.5% lower latency at higher accuracy

More from Models