Position Bias Undermines Preference Consistency in Listwise LLM-Based Reranking
Ethan Bito, Yongli Ren, Estrid He
cs.IR
2026-08-04
In recommendation, the candidate set is unordered, yet an LLM reranker produces different rankings when the input order changes. Rather than tracking only final-list changes, this work treats rankings under permutations as an "induced preference system" and measures position sensitivity at the pairwise, global, and listwise levels. The three measures agree closely but diverge from effectiveness and marginal exposure: the most accurate method is the least stable.
In two-stage recommendation, LLMs are often used as listwise rerankers: a retriever proposes candidates, and the LLM reranks them using user history and item metadata. But there is an overlooked reliability problem: the candidate set is fundamentally unordered, yet feeding the same set to an LLM in different orders can yield different rankings. That violates the basic assumption that a ranking should not depend on arbitrary input serialization.
Prior work calls this position bias and offers mitigations (permutation aggregation, calibration, sequential selection). But evaluation mostly tracks changes in the final ranked list or marginal exposure per input position, without examining how candidate-order sensitivity shows up in the preference structure the LLM induces. The questions here: when candidates are reordered, are local pairwise preferences stable? Can the global preference be explained by a single ordering? And how do these relate to ranked-output consistency, recommendation effectiveness, and marginal exposure?
The authors treat the multiple rankings produced by permuting the same candidate set as observations of an "induced preference system," and define three consistency measures characterizing candidate-order sensitivity at the local, global, and output levels.
Pairwise Preference Instability (PPI): does the relative preference between a pair flip depending on which position buckets (head/middle/tail) they land in? For each pair, take the max difference in preference probability across bucket combinations, then average over pairs. Global Preference Inconsistency (GPI): can the induced pairwise system be explained by a single global ranking? A weighted Kemeny aggregation finds the minimum-disagreement ranking, and GPI is that disagreement mass. Listwise Output Consistency (LOC): agreement between complete rankings from different permutations, measured by Kendall's τ. There is also marginal position-exposure bias: the probability of each input position reaching the top-k, ideally uniform (k/K).
PPI, GPI, and LOC are not independent; they are three levels of the same phenomenon: local, global, and observable output.
Two datasets (MovieLens-32M, Amazon Books), three models (Llama-3.2-3B, Mistral-7B, Qwen2.5-7B), K in {15, 25, 50}, 20 permutations per query, rankings extracted from token log-probs (deterministic). Compared are zero-shot reranking and three mitigations: bootstrapping (Borda), SGS (sequential greedy selection), and STELLALW (calibration-based).
The core finding: PPI, GPI, and LOC agree closely, producing the same method ordering across all model×dataset combinations. But they diverge from recommendation effectiveness and marginal exposure bias.
| Metric (MovieLens-32M, Llama-3B, K=25) | Zero-shot | STELLALW | SGS |
| HR@5 ↑ | 0.556 | 0.592 (highest) | 0.563 |
| PPI ↓ | 0.478 | 0.833 (worst) | 0.197 (best) |
| GPI ↓ | 0.115 | 0.365 (worst) | 0.059 (best) |
| LOC ↑ | 0.636 | 0.118 (worst) | 0.827 (best) |
The most counterintuitive point: STELLALW gets the highest HR@5 yet has the highest PPI and GPI and the lowest LOC. The most effective method has the least stable preference structure. It flattens marginal exposure across positions but does not buy stable pairwise preferences or a coherent global ordering. SGS is best on both consistency and exposure but needs K sequential passes (25 when K=25), the most expensive; bootstrapping is moderate on consistency (3 passes); STELLALW needs up to 10 inference passes plus a 150-probe offline calibration.
The conclusion is direct: fixing marginal exposure alone is not enough to give an LLM reranker stable pairwise preferences and consistent rankings. LLM rerankers should be evaluated on effectiveness, permutation consistency, marginal exposure, and inference cost together.
This is a sober reliability warning for anyone using LLMs as recommendation rerankers. Many watch HR/nDCG and assume a higher score means a stable ranker, but if reordering candidates changes the ranking, the ranking function is not well-defined. The three measures (PPI/GPI/LOC) are operational diagnostics, and the code is open (InvariRank). For production, it surfaces a practical trade-off: the most accurate method (STELLALW) is the least stable, while the most stable (SGS) is the most expensive. There is no free lunch.
This is a 5-page RecSys short paper; the contribution is an evaluation framework and diagnostic findings, not a new model. All three measures rest on sampling permutations, so the permutation count (M=20) and bucket scheme affect the numbers; PPI uses buckets rather than exact positions as a sparse-data compromise. The global-ranking solve is approximate (local search), not exact Kemeny-optimal. Mitigation effects are validated on only two datasets and three models, so generalization is limited. It does not offer a fix that is both high-effectiveness and high-consistency, only flags it as open.