Position Bias Undermines Preference Consistency in LLM Reranking

_reachsumit · x · 2026-08-05

This paper reveals that listwise rerankers based on Large Language Models (LLMs) suffer from position bias, yielding unstable rankings when faced with equivalent candidate permutations. Although recommendation candidates form an unordered set, decoder-only LLMs allow input order to affect model scoring and pairwise preferences.

The researchers introduced a novel evaluation framework measuring candidate-order sensitivity across three levels: pairwise preference instability, global preference inconsistency, and listwise output consistency. Experiments demonstrate that simply improving relevance or flattening exposure bias does not necessarily restore stable pairwise preferences or globally coherent ranked outputs.

Original post →

Position Bias Undermines Preference Consistency in LLM Reranking

More from Research