There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items
V. S. Raghu Parupudi
cs.AI
2026-07-17
Twelve open models answer 3,679 MC items under 26 defensible harnesses. Adjacent models tie on stable items; fragile items carry 95.7% of each gap.
Multiple-choice benchmarks fix the questions and the gold answers. They do not fix option order, prompt wording, or whether the answer is parsed from generated text or taken from per-option likelihoods. Prior work reports that sensitivity as variance on the aggregate score. That leaves two questions open: which items move, and are they the items that separate one model from the next?
Noise spread evenly across a test is a different object from noise that lands on the items that rank neighbors. The two look the same at the score level. They have opposite consequences for a leaderboard.
The evaluation harness is treated as an independent variable, resolved to single items. The instrument is a fragility grid. Twelve instruction-tuned open-weight models from Qwen3, Llama 3, Gemma 4, and Mixtral, 3B to 70B, answer the same 3,679 four-option items under 26 configurations. Items come from ARC, HellaSwag, MMLU, and TruthfulQA. Decoding is greedy and zero-shot. Items, weights, and decoding stay fixed. Only the harness moves.
Three axes: option order (original plus five fixed shuffles), prompt format (letter-period, parenthesized letters, digit labels, an explicit letter instruction, plus two order-invariant cloze stems), and scoring (parsed generation versus length-normalized option likelihood). Twenty-four generation setups plus two likelihood cloze stems give 26. The reference is letter labels, original order, generation scoring. None of the 26 is adversarial.
An item is robust-correct for a model if all 26 configurations get it right, robust-wrong if all get it wrong, and config-fragile if it flips. Each adjacent-pair gap then splits into jointly stable mass and fragile mass.
A model's score is a band. gemma4-31b sits at 88.0% on the reference, 30.6% at the low configuration, and 88.6% at the high one, a 58.0-point spread, matching the abstract's 31 to 89 percent. Across the roster, 85% of reference-correct answers can be flipped to wrong by another defensible setup. Robust accuracy is at most 21.5% for any model and below 7% for five of them. The widest band belongs to the reference champion, not to a weak model.
On items both models in an adjacent pair answer stably, 5 of 11 pairs match item for item, 4 differ on a single item, and 2 cancel in both directions. All 10 pairs with a nonzero full-set gap are manufactured: ordered on the full set, tied on the robust set. Fragile items carry 95.7% of a pair's gap on average, 99.7% when items are pooled. gemma4-12b and qwen3-14b reverse under 21 of 26 configurations. Mean Kendall correlation with the reference ranking is 0.42, so a typical configuration flips about 29% of the 66 pairwise orderings.
The harness selects the winner. Four of 12 models reach rank one under some configuration; eight reach the top three. gemma4-31b wins 16 setups by a median of 181.5 items; qwen3-14b wins 8 by a median of 98.5. The other two champions each win once, by 1 item and by 3. mixtral-8x7b is 11th under the reference and first under likelihood scoring. Swap letter labels for digits on a matched stem and the winner flips under every option order.
Compression keeps the fragile items. Item discrimination correlates with fragility at 0.28 (95% CI 0.25 to 0.30). The 100 most discriminative items have mean fragility 0.96 against 0.85 overall. Cutting to 100 items raises the champion count from 4 to 5.1 for a random subset, and to a mean of 5.7 when items are chosen by discrimination.
Scoring does the damage. Holding the other axes at the reference, option order preserves 0.60 of credited accuracy, prompt format 0.40, and generation-versus-likelihood 0.31. All three together leave 0.11. The axis that protocols most often pin down is the one that matters least.
A single leaderboard number is a mixture of model and harness. Adjacent ranks do not exist on the items both models answer stably. Four reporting changes follow: publish a band with robust accuracy; compare models on jointly robust items; disclose the harness in full, scoring first; check a compressed benchmark for configuration stability before trusting it.
The paper does not claim these models are equal, and it does not claim multiple-choice tests carry no signal. It claims that the adjacent order printed by one harness is not identified by the items those models answer stably.
The roster is 3B to 70B open instruction models on four-option items. No frontier closed models, no open-ended generation, code, or agents. TruthfulQA is assembled into four options with gold always first in the native order; dropping it cuts the champion count from 4 to 3 while the main claims hold. Jointly robust sets are 72 to 336 items, so the tie is a small-sample statement, not a proof of equal ability. Decoding is greedy, so the grid shows that harness variance alone can manufacture an order; it does not compare that variance with sampling. The scoring axis is two cloze likelihood setups, not every likelihood implementation.