S1CogBench: 24 cognitive experiments find System One decision models act well, gather evidence poorly

S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments

Dengzhe Hou, Kazunori D Yamada

cs.CL, cs.AI, q-bio.NC

2026-10-31

S1CogBench tests 8 System One decision models on 24 cognitive experiments: Nox leads 18 targets, yet four models stop instantly in information search, and high autonomous scores do not predict human choices.

What problem this solves

Jev, Laya, and Intern-Decision exemplify a shift: instead of generating free text, one call returns a single typed decision that software can execute directly. These models already sit inside routing, guardrails, and automated gates, and a small one can be invoked thousands of times inside a larger system. A typed interface guarantees the output is executable, not that it used the right evidence: a model can return a clean distribution while ignoring feedback or stopping its search early, and no generated text is left to inspect. Closed-loop evaluations show such small models mostly fail, but cannot say which cognitive demand broke. Cognitive experiments can, because they manipulate information, uncertainty, memory load, and future opportunities one at a time, while existing cognitive benchmarks cover language and multimodal models only. S1CogBench is the first to run System One decision models through established cognitive experiments.

Method

All 24 tasks come from published experiments (WCST, Horizon, RLWM, Mouselab, two-step, and others) grouped into ten domains; WCST, Horizon, and Mouselab each run in both a human-prediction and an autonomous mode, for 27 evaluation modes. At each decision point, an adapter supplies the visible state, the history, and the ordered legal responses, and the model returns a distribution over them. Three choices give the benchmark its resolution:

Baselines are Uniform (equal probability over legal responses) and a task-specific Reference (models fitted to human data or analytic solutions). Seven open models run all 168 task-model combinations: four System One decision models (Laya, Intern-Decision-4B, Nox-4B, CLM-8B) plus Qwen3.5-4B, Llama-3.1-8B, and Minitaur-8B, fine-tuned on Psych-101 human behavior. The Jev API runs all 27 modes; Claude5.5 runs a subset due to cost.

Results

The spread is wide. Under original readouts, targets where a model fails to beat Uniform: 17 for Laya, 13 for CLM, 11 for Llama, 7 for Minitaur, 6 for Qwen, and 3 each for Intern-Decision and Nox. Nox leads the open models on 18 of 26 targets. Intern-Decision beats its own Qwen3.5 backbone reliably on 19 targets and loses on 2, so fine-tuning for structured decisions buys real capability.

Acting well and predicting humans come apart:

ModeAutonomousHuman NLL
Nox, WCST32.1% (best open; humans 50.0%)1.552, worse than Uniform's 1.386
Minitaur, WCST25.2%1.344, best open
Qwen, Mouselabreward 17.40, near 17.55 for revealing all2.464, worse than Uniform's 1.634

Jev reaches 89.5% in RLWM under human-block histories, close to the same-participant fitted reference at 89.3%, but it reports probabilities to two decimals, which yields infinite NLL on five of nine prediction targets.

Information gathering is the most common failure. In Mouselab, Intern-Decision, Laya, Minitaur, and CLM choose STOP on the first move in all 23,750 episodes for zero reward; Nox and Llama reveal one node and quit. STOP is always the first listed option, and the authors concede a position bias is not excluded.

Sensitivity to changing demands runs consistently below humans. When Horizon goes from 1 to 6 remaining choices, humans pick the less-observed arm 10.00 percentage points more often; the eight models move by -0.49 to 2.62 points. In RLWM, Nox reaches 86.30% accuracy but loses only 4.84 points from set size 2 to 6, against a human drop of 12.74, while Intern's learning curve gains 61.49 points over ten presentations, above the human 46.35: the supplied history gets used, just not the way people use it. In the restless bandit, Minitaur chooses the same arm on all 800 trials while its reported probability swings between 0.56 and 0.93. The WCST floor is not about text format: Claude with adaptive thinking scores 63.51% to 66.22% on 74 shared decisions where open models get 18.92% to 43.24%, so the task does permit learning from feedback in text form.

Latency is practical. Median free-text generation in Horizon takes 2.958 s for Llama and 16.330 s for Qwen, against 36.97 and 51.66 ms for direct decisions, 80x to 316x slower; Llama's free-text WCST is also worse (25.9%, with 2,217 invalid responses, versus 31.4% direct). Matched one-token generation costs only 0.24% to 2.02% more than a direct decision, and size-matched Intern-Decision shows no speed edge over Qwen3.5: structured fine-tuning buys no speed and pays no latency tax.

Why it matters

Before wiring these models into automated decision systems, this is the checklist: high autonomous scores do not show that a model gathers information, adapts to feedback, or predicts what people choose. Three usable takeaways: structured-decision fine-tuning pays (Intern beats its backbone on 19 of 26 with no speed cost); the readout is part of the deployment, since swapping action-extraction rules reverses rankings, so lock it down together with the model; and pay for generation only when you need prose, because direct decisions are two orders of magnitude cheaper. For researchers, cognitive profiles built from condition effects and learning curves localize what a training objective should fix better than any single score.

Limitations

The authors' own list: 24 tasks and five System One models; text conversion and other adaptations; architecture, capacity, training data, and input retention vary together, so nothing attributes cleanly to architecture; Minitaur's fine-tuning data shares source experiments with Horizon and eight Psych-101-derived modes, making its prediction scores in-distribution; API models differ in every dimension. Two more from a close read: Nox is fine-tuned from Qwen3.5-4B-Base while the evaluated Qwen3.5-4B is posttrained, so that pair is not a backbone comparison and the fine-tuning claim rests on a single pair. Interface engineering and degenerate behavior also pollute scores: Laya's SDK rounds probabilities to four decimals, assigning zero to 192 human choices, and Minitaur answers 50% on 319 of 330 nonliteral-number items.

Terms

Source

Related papers

All paper explainers