Thinking effort aligns between humans and reasoning models in abductive reasoning
Henry Arthur
EMNLP 2026
cs.CL, cs.AI
2026-09-02
On 160 abductive items, LRM tokens track human RT after length controls. GPT-OSS-120B greedy r=0.552; averaging recommended-temperature samples lifts GPT-OSS-20B 0.41 to 0.55.
de Varda et al. (2025) reported that chain-of-thought token counts in large reasoning models track human reaction times across several reasoning tasks. Skeptics answered on two fronts. Traces may not be faithful to the computation. And on syllogisms, difficulty is partly readable from form, so a model could spend more tokens on items that look hard without searching.
This paper moves the test to abduction, Peirce's "best explanation for underdetermined premises." Wet grass could be rain, a neighbor's hose, or an overflowing well. There is no syllogistic mood that marks which item is hard. Deductive difficulty can leak from schema; abductive difficulty comes only from what each hypothesis would explain. The task is forced-choice inference to the best explanation: two observations, two hypotheses, pick the one that covers both. Items come from AI2's ART, adversarially filtered with BERT so the pair is hard to split.
120 native English speakers in the US (127 recruited on Prolific, 7 dropped for implausible timing) saw 160 ART test items in four non-overlapping blocks, answering with Q and P. Mean completion time was 9.2 minutes. Hypotheses A and B differed by at most one word, so length was a poor cheat. Human accuracy was 78.8%. Mean total word count was 33.5.
Models follow de Varda: thinking-trace tokens as cost, partial Pearson with mean human RT, both log-transformed, residualized on prompt token count from each model's tokenizer. Main setting: greedy decoding, default reasoning effort. The panel is DeepSeek-R1, Qwen3-235B/32B Thinking, GPT-OSS-20B/120B, GLM-4.5-Air, Kimi-K2 Thinking, plus DeepSeek-V3 as a non-reasoning baseline (step-by-step appended; all generated tokens count).
R1, GPT-OSS-20B, and Qwen3-32B were then sampled at provider-recommended temperatures (0.6 or 1.0). Item-level mean tokens and majority-vote labels were pooled. Convergence meant a 95% interval on partial r narrower than 0.03 and a mean shift below 0.01 when adding a run. K=10 for R1, 25 for the other two. The same K was repeated at temperature 2. Every token–RT correlation controls for prompt length, because length correlates with human RT at r=0.669.
Under greedy decoding, every model correlates with human RT after the length control.
| Model | Partial r |
| GPT-OSS-120B | 0.552 |
| GPT-OSS-20B | 0.41 |
| Qwen3-235B Thinking | 0.359 |
| DeepSeek-R1 | 0.327 |
| Kimi-K2 Thinking | 0.303 |
| GLM-4.5-Air | 0.263 |
| Qwen3-32B Thinking | 0.229 |
| DeepSeek-V3 (no native reasoning) | 0.197 |
The ensemble (mean log tokens per item) is r=0.42, close to de Varda's within-task syllogism result (0.43) and above relational reasoning (0.27). R1 versus V3 is not significant (Fisher z=1.23, p=0.217). Humans score 82.5% on items R1 got right and 47.8% on items it got wrong, below chance. Model-consensus accuracy versus human accuracy is partial r=0.668. When at least 75% of models fail, humans sit at 48.7%; when at least 75% agree on the right answer, humans sit at 84.6%. The split-half noise ceiling is r=0.861. GPT-OSS-120B's R²=0.303 on residual RT is about 35% of that ceiling. Size does not order alignment: 20B GPT-OSS is second, well above 671B R1 and 1T Kimi-K2.
Averaging stochastic runs raises alignment. GPT-OSS-20B goes 0.41 → 0.55 (Steiger–Williams p<0.001), stable by 19 runs; Qwen3-32B 0.23 → 0.32, stable by 18; R1 0.32 → 0.37, not a significant lift, stable by 8. At temperature 2, GPT-OSS-20B falls to 0.22 and has not converged by 25 runs. Error alignment is also strongest at the recommended temperature: GPT-OSS-20B's point-biserial with human accuracy is 0.51 at T=1 versus 0.42 greedy.
Raising GPT-OSS-20B's reasoning effort helps the hardest tercile (69.2% vs 53.8% at low effort) without much change in token–RT alignment (0.34 low, 0.30 high).
Abduction does not hand the model a format cue for difficulty, and token cost still tracks human RT. Humans and models also fail similar items. That is a harder setting for the claim that CoT length can stand in for thinking cost. The practical note for evals: the temperature on the model card, averaged over enough runs, aligns better than a single greedy sample; creative T=2 scatters the correlation. Parameter count is not the driver on this commonsense set; 20B GPT-OSS beats 671B R1.
Alignment is not mechanistic identity. The authors keep the claim at "shared constraints as an empirical fact," not "brains implement Transformers." GPT-OSS-120B still only takes 35% of the noise ceiling.
One abductive format: forced-choice best explanation, not hypothesis generation, not long-form detective reasoning. Main runs use default effort, not difficulty-matched effort; the appendix shows high effort helping only the hardest tercile, so ceiling effects may compress the headline rs. Prompts mirror human instructions, but models are prompt-sensitive. ART contamination is not ruled out; shared errors make pure memorization less likely, they do not prove a clean pretraining history. The R1–V3 gap is not significant, and commonsense abduction is exactly the setting where world knowledge in a base LLM can fake a weak correlation without an explicit trace. CoT-as-cost still faces the "performative scaffolding" critique. Multiple decoding settings probe robustness; they do not close the mechanistic hole.