Accuracy and Order Sensitivity Diverge Under Label-Free Strategies
Karl Hanna, Chen Feng
cs.CL, cs.AI
2026-08-12
Across 6 models and 2 benchmarks, label-free MCQ strategies fail: two-stage prompting drops accuracy in 11 of 12 pairs, with the bottleneck traced to withholding options; removing positional bias does not buy accuracy, while cyclic permutation improves 5 of 6 models per benchmark.
An MCQ benchmark score conflates two things: whether the model knows the material, and how sensitive it is to the order of the options. Rotate four options and the answer flips; such scores are shaky as a measure of knowledge. A family of "label-free" fixes proposes hiding the A/B/C/D labels while the model commits, removing positional influence and supposedly cleaning up the score. This paper tests that intuition systematically and finds it does not hold: eliminating positional influence does not reliably buy accuracy.
The result matters because label-free strategies keep circulating as cheap debiasing for evaluation.
Two label-free strategies, six models (GPT-4.1 mini, Gemini 2.5 Flash, Llama 3.1 8B, Qwen 2.5 7B in two deployments each), MMLU (50 subjects x 20 questions = 1,000) and ARC-Challenge (1,000), temperature 0:
The key addition is a 2x2 decomposition crossing stage-one option visibility (hidden/visible) with the matcher (LLM/embedding), which isolates where two-stage prompting fails. Order sensitivity is measured two ways: flip rate (whether cyclic permutations of one question yield at least two distinct semantic answers, a direct per-question measure) and RStd (the standard deviation of recall across the four answer positions, an aggregate measure).
Neither strategy reliably improves accuracy. Two-stage prompting lowers accuracy in 11 of 12 model-benchmark pairs; independent hypothesis scoring lowers it in 8 of 11 valid pairs.
| Configuration (example) | MMLU | Note |
| GPT-4.1 mini baseline | 81.8 | |
| Two-stage (hidden + LLM matcher) | 80.1 | zero parse failures, still drops |
| Hidden + embedding matcher | 45.8 | cliff |
| Visible + embedding matcher | 81.7 | most of the loss recovered |
| Visible + LLM matcher | matches baseline | the only configuration that consistently does |
| Cyclic permutation (same model) | 83.0 | within noise, direction up |
The decomposition shows the bottleneck is withholding options in stage one, not the matching step: with options visible, even an embedding matcher climbs from 45.8 back to 81.7. Gemini's two-stage drop from 84.9 to 68.3 on MMLU (96.9 to 86.6 on ARC) is mostly parse failures, a reminder that the fragility is deployment-dependent.
Against intuition, cyclic permutation (four rotations with majority vote) improves 5 of 6 models on both MMLU and ARC, with Llama-local going 58.0 to 72.9 on ARC (+14.9 points). It never removes positional influence, only averages over it, and it works. Order sensitivity and accuracy are decoupled: GPT-4.1 mini's MMLU flip rate roughly halves from 21.6 to 11.8 while accuracy falls from 81.8 to 80.1. Two-stage prompting increases flip rate in 7 of 12 pairs and RStd in 5, so it does not even deliver reliable debiasing.
For anyone building benchmarks or running evaluations, two convenient paths are crossed off. Generate-then-match systematically underestimates models and should not be used as a debiasing wrapper; if a two-stage pipeline is unavoidable, options must stay visible in stage one. A position-insensitive model is not a stronger model; report the two dimensions separately. Cyclic permutation costs k calls per question but averages out position while aggregating samples, and it is the closest thing to cheap-and-effective in the results, with the largest gains for small open models (+6.8 and +14.9 points for Llama-local). The ChoiceBench code and question sets are on GitHub.
Stated by the authors: six models, two benchmarks, four-option settings only; each condition run once, so provider-side non-determinism between runs is untested; the flip-rate experiment lacks repeated identical-order controls, so some flips may be serving noise; two-stage prompting is evaluated in a single instantiation without stage-one reasoning and without a stronger matcher; several cells score far fewer than 1,000 questions due to parse failures (Llama-local MMLU: 688/1,000), so flip rates and RStd reflect subsets. Sampling 20 questions per MMLU subject also leaves per-subject variance under-sampled.