DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems
Xialie Zhuang, Mingcong Lei, Zunhai Su, Ri Liu, Jianjin Li, Peixian Ma, Jia Pan, Ngai Wong, Guanhua Chen, Chuan-Wei Kuo
cs.AI
2026-10-21
DecisionPivot scores 40 decision models on 7,800 verified contrast items. Two 27B models 2.6 public-benchmark points apart reach 70.1% and 44.2% contrast-group accuracy.
A decision model does not write the reply. It reads a state, a question, and a short list of options, and returns a probability for each option in one forward pass. Routing, guardrails, output checks, and agent goal tests all run this way. Uncertain cases go to a larger model.
Accuracy on public sets hides two problems. On llmbar, a rule that ignores the text scores 73.0%, against a 50% chance baseline, so surface cues already move the number. A contrast pair can also change one decisive fact and leave the rest fixed. Yes on both items means 50% item accuracy and a failed group. Training notes usually name a data source, not whether a given item was included.
DecisionPivot uses one interface on two tracks. Each item is a state, a typed question, and an ordered option list. The prediction is the option with the highest probability. A missing prediction counts as wrong.
The core track is generated: 13 families, 600 items each, 7,800 items, 200 at each of three difficulty levels. A generator samples a structured case, computes the answer, then edits one decisive detail so the correct option flips and the distractors stay. Threshold cases sit on the boundary. One cent of shipping fee is enough to cross a price limit. policycase works differently: distinct outcomes share one policy, and there is no flipped twin. A second solver, written by someone who had not read the generator, matches all 600 gold labels in every family. Every core item had at least two human reviewers. Probes that see only the options or the length land near chance. A schema-only check still scores 91.7% item accuracy on toolcallvalid.
Seven roles run from the incoming request into the data pipeline: answerability and routing, policy checks, output verification, tool-call and goal checks, clause edits, dependency resolution, ledger reconciliation, and constraint matching. The reference track converts 35 public test sets, 32,233 items, and keeps each original metric. Scores are chance-corrected, then averaged over the seven layers. Combined, that is 48 tasks and 40,033 items. Overlap filtering removed 3,016 reference items, 8.5%, found in the authors' own 27B training files. Other models get only a source-level flag.
Pair accuracy counts a contrast group as correct only when every member is correct. On a yes/no pair, two yes predictions score 50% item accuracy and 0 pair accuracy. Uniform guessing lands at 39.6% item accuracy and 16.1% pair accuracy. Pair direction only checks whether the positive member gets a higher probability of yes than the negative member. Both values can still sit above 0.5. Coverage at 5% walks items from highest confidence downward and keeps the longest prefix whose empirical error stays at or below 5%. That figure is retrospective, with a separate cutoff per family.
Forty decision models were run through their documented interfaces. Core scores in the table are computed per family and then averaged. Eleven open general models and three API chat models were also run. The top two rows are SL-27B from StartLux, the group that built the benchmark. AutoThink writes a reasoning trace only when the best one-pass probability is below 0.9.
| Model | Reference (0 = chance) | Pair accuracy | Item accuracy | Coverage at 5% error |
| SL-27B AutoThink | 58.9 | 86.9% | 94.0% | 93.8% |
| SL-27B Instant | 59.4 | 75.4% | 86.4% | 79.6% |
| Decision 3.0 d3 | 56.7 | 71.3% | 83.9% | 75.1% |
| Perplexity Decider v1.1 | 57.0 | 70.1% | 83.4% | 72.3% |
| JEV-27B | 54.4 | 44.2% | 70.3% | 34.8% |
| Kev 27B | 54.0 | 44.2% | 70.6% | 35.2% |
| Uniform guess | 0 | 16.1% | 39.6% | not reported |
Perplexity Decider and JEV-27B score 57.0 and 54.4 on the reference track, and 70.1% and 44.2% pair accuracy on the core track. Across eight models the rankings move together (Spearman rho 0.88), while the spread is 11 reference points against 74 points of pair accuracy. Nineteen of the 40 decision models fall below the 16.1% chance line on pair accuracy, and 30 do so at difficulty 3. Missing predictions from input limits are included. Decider itself is uneven: 97.5% pair accuracy on policycase, 33.8% on the five-option triageorder task.
On 2,400 binary pairs, Decider puts a higher probability of yes on the positive member in 92.3% of pairs and answers both members correctly in 74.1%. Clef 27B scores 78.6% on direction and 25.0% on pair accuracy. In the shipping-fee pair, 24.03 versus 24.05 SEK around a 500.84 limit, Decider returns 0.86 and 0.84: the order is right, and both answers are yes. A single global threshold fit on these same pairs lifts the two models only to 74.2% and 30.6%.
Kev 27B and JEV-27B match on item accuracy, 70.6% and 70.3%. At confidence of at least 0.90, Kev is 93.8% accurate on 34.1% of items; JEV is 88.9% accurate on 47.4%. Family-averaged coverage at a 5% error target is 35.2% and 34.8% for those two, against 79.6% for Instant and 72.3% for Decider.
From difficulty 1 to 3, pair accuracy falls from 87.7% to 57.4% for Decider, from 90.7% to 65.3% for Instant, and from 89.2% to 82.2% for AutoThink. On the total-price and minimum-discount contrasts in the figure, four one-pass models reach at most 5% pair accuracy, against 57% and 76% for AutoThink. On current-time contrasts in triageorder those four score 0%, and AutoThink scores 85%.
On a separate 1,296-item subset, GLM-5.3 reaches 93.5% pair accuracy (95% interval 91.7 to 95.3) at the higher reasoning setting and 81.5% (78.4 to 84.6) at the lowest. Reasoning length rises from 121 to 265 tokens per item. The lowest setting still includes a hidden trace. On that subset AutoThink scores 85.9%, Instant 75.6%, and Decider 70.4%. On the full tracks AutoThink has 86.9% pair accuracy and a 58.9 reference score; Instant has 75.4% and 59.4. Gains on contrasts do not lift the public aggregate.
A router or a guardrail has to change its decision when the decisive fact changes, and its high-confidence calls have to be right. A gap of about two points on a public index shows neither.
Among incorrect predictions, confidence of at least 0.90 accounts for 14.6% of Instant's errors and 33.2% of Decider's. Guardrails should weight the second number. On cases with no planted defect, Kev scores 56.5% and Clef 27B scores 32.7%. With a planted defect they score 77.5% and 76.7%. Decider scores 80.9% on both. False rejects and false accepts will not land on the same side.
SL-27B and DecisionPivot come from the same team. Instant's reference score of 59.4 is 2.4 points above Decider, paired 95% interval 1.6 to 3.2. The point estimate is higher on six of seven layers, and some layer intervals include zero. The pair-accuracy lead is not an independently replicated ranking.
The cases are written items, not a live system. Multilingual behavior and speed are out of scope. Difficulty raises length and rule complexity together, so the drop at level 3 isolates neither. Coverage at 5% is retrospective, and a deployment cutoff needs its own calibration. A redraw of 4,200 items from the same generators preserves rank for 34 models (Spearman rho 0.994), and the score-change interval includes zero for 32 of them. That is stability inside one generator design. The paper says a third-party evaluation is still needed. The GLM-5.3 comparison also changes the sampling setup: the higher setting uses the provider default and a larger output budget, and the lower setting uses temperature 0.
The 19 models under the chance line include input-limit failures, so the bottom of the table is partly an interface result. On version-1.0 wording variants, pair accuracy moves only between -2.2 and +2.2 points, while natural wording changes up to 20.7% of predictions. A second AutoThink run scores 87.2% against 86.9% and changes 5.6% of item answers. Stable totals with unstable items were not rerun on the revised 1.1 set. The reference track still includes llmbar, where a content-blind rule scores 73.0%.