More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models
Tianxiang Gao, Jinzhe Li, Zhiyuan Li, Yi Chang, Yuan Wu
cs.AI
2026-09-30
JEV-like direct-decision models use only 67-76% of the effective ordinal scale across 36 datasets; model size doesn't help, but targeted BA-LoRA post-training restores it to ~86%.
Direct-decision models skip text generation: the candidate set ships with the input, and the model returns a probability over candidates plus an argmax decision. JEV 1.13 is the hosted version of this interface; the KEV family (0.8B/4B/9B) is an open reimplementation on Qwen3.5 backbones. Structured output and low latency make them natural classifiers and automatic evaluators. The catch is that accuracy only records item-level correctness. A model can score well while barely touching most of the scale it was handed.
The paper starts from ANLI. JEV 1.13 reaches 74.95% accuracy on 6,400 dev and test items, yet sends 38.8% of all predictions and 51.3% of its 1,603 errors to Neutral, even though the three gold labels are nearly balanced (2,140 / 2,136 / 2,124) and Neutral holds no fixed candidate position. ANLI is nominal, so this is a clue rather than proof of ordinal compression. The risk holds either way: plausible accuracy and a systematically distorted output distribution can coexist. The authors name the broader pattern ordinal scale-utilization bias.
The measurement is entropy-based effective support. Exponentiating a distribution's entropy gives the effective number of labels it actually uses; dividing the predictions' effective support by the gold labels' yields the utilization ratio R, where 100% means equal coverage and TVD tracks distributional shape separately. This separates scale use from four nearby explanations: low accuracy, gold imbalance, position bias, and candidate count.
Four experiments:
| Setting | Metric | Result |
| 36 ordinal datasets | Gold-relative utilization R | 67.2% (KEV-4B) to 75.8% (JEV 1.13); TVD 29.8-37.5% |
| 4 nominal datasets (7-14 classes) | R | 86.6%-101.9% |
| RealToxicity, balanced 14 levels | R | 25%; JEV piles 66.2% of predictions on Level 01 |
| KEV 0.8B/4B/9B | Ordinal accuracy / R | 28.8%-33.4%-35.0%; R 68.9%-67.2%-73.2%, non-monotonic |
| After candidate-order randomization (ordinal) | R | 67.7%-73.9%, still below the 98.6%-99.3% of nominal tasks |
| K=2 to 14, same items, equal-frequency | R | JEV 95.9%-54.5%, KEV-0.8B 77.3%-26.5%, KEV-9B 96.1%-74.9% |
| BA-LoRA on 8 training sources | R / accuracy | KEV-0.8B 47.4%-86.8%, KEV-4B 46.4%-85.6%; accuracy +13.0 and +15.3 points |
Nominal tasks are not fully immune: KEV-0.8B sends 55% of its SCOTUS predictions to the broad Miscellaneous class, for R = 51%.
The order experiment has a side finding worth flagging: two random orders agree on only 58.1-59.9% of ordinal items for the KEV models and 74.4% for JEV 1.13, versus at least 95.6% on nominal items, while macro accuracy moves by at most 0.55 points. Individual decisions are highly order-sensitive; aggregate accuracy hides it completely.
The K experiment's sharpest finding is the split between probabilities and decisions. For KEV-4B and KEV-9B, soft utilization Usoft stays above 96% at every K, so all levels remain represented in the average probability vector, while the top-1 margin collapses from 50-54 points at K=2 to 4-6 points at K=14. Coarse ordering survives; the decision boundaries between adjacent levels collapse. The DBpedia reference holds R near 100% at K=14, ruling out candidate count alone as the cause.
Post-training lifts KEV-4B from R 67.2% to 82.3% across all 36 ordinal datasets, with QWK rising from 0.456 to 0.606. Transfer splits by size: on the 28 unseen sources, 4B improves from 73.2% to 81.3% while 0.8B drops from 75.0% to 72.7%.
If a model serves as your grader, rubric judge, or first stage of an evaluation cascade, accuracy is half a quality signal. The paper offers a cheap audit recipe: report R and TVD next to accuracy, run several randomized candidate orders, and trace utilization as the scale is refined. Three steps expose levels that were offered but never used. For model builders, the BA-LoRA result shows compression is a learned decision-boundary problem that targeted data fixes without touching the architecture. To be clear about the framing: this is a measurement study that names a bias, quantifies it, and shows it is repairable. No new architecture.
The authors list several: the ordinal/nominal split is a binary operationalization with no graded measure of semantic coupling; the same-item refinement covers only three score sources; the DBpedia reference sits near ceiling for every model rather than being difficulty-matched; the position experiment uses two permutations per item; R = 100% does not imply mass lands on the same labels, which is why TVD is reported alongside. One hosted version and one backbone family are covered, so transfer to other direct-decision models is untested. There is also an unusually frank disclosure: SST5, DBpedia-14, and IMDb appear among KEV's training sources, item-level overlap was not audited, and KEV numbers on those three datasets may partly reflect training exposure.
Two things stand out on a close read. Macro accuracy on the 36 ordinal datasets is only 28.8-37.7%, so the models largely fail these tasks to begin with; gold normalization blocks the label-imbalance explanation but not the weak-capability one. And the recovery on training sources is what supervised training does by construction; the real evidence for repairability is transfer, which holds at 4B and reverses at 0.8B. That looks like capacity plus data coverage, not general debiasing.