One Bias Score, Two Factors: POSTECH Paper Decomposes Option Bias in MC Evaluation
Hyunjong Ok · hf · 2026-10-16
A POSTECH paper (arXiv, Oct 14) unifies selection bias and order sensitivity in multiple-choice LLM evaluation via the pooled selection-rate standard deviation, showing it equals the product of two label-free factors: the flip rate (fraction of questions whose chosen option changes across orderings) and positional skew (SD of option-letter selection rates).
Across 41 checkpoints and 12 benchmarks:
- Flip rate tracks accuracy and output uncertainty; positional skew tracks the option-symbol prior
- Prior correction mainly reduces positional skew; aggregating each ordering with its reversal mainly reduces flip rate
- Both narrow the accuracy range across correct-answer letters, but aggregation leaves fewer model-rank reversals
- Applying prior correction before aggregation reduces both factors
Takeaway: the two mitigations target different factors and should be reported alongside accuracy.
More from Research
- Nature paper uses language models to map single neurons behind human language — raphaelmilliere · 2026-10-16
- Reviewer shares workflow: feeding the finished review — never the paper — to AI for a second pass — StingMeleoron · 2026-10-16
- 3SUM record falls to O(n^1.921), Lean-verified — but will it yield usable algorithms? — iskander · 2026-10-16
- Ablation studies to help Kaggle Gemma 4 competition participants — ayushthakur0 · 2026-10-16
- Experts put global AI catastrophe risk before 2100 at 5%, LEAP survey finds — ben_j_todd · 2026-10-16
- Historical lesson for AI: tractors automated farm labor, teen boys stayed in school longer — soumitrashukla9 · 2026-10-16