Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, Quanquan Gu
ICML 2025)
cs.AI, cs.CL, cs.LG
2024-10-03
Tsinghua/UCLA's GPM embeds replies as skew-symmetric vectors with O(K) queries, scoring 100% on cyclic prefs and lifting Gemma-2B RewardBench from 74.85 to 82.29.
Preference modeling for LLM alignment sits on two poles. A Bradley-Terry (BT) reward model scores each reply with a scalar, so ranking K candidates costs O(K) forwards. That cheap query comes from an assumption: preferences are transitive. If A beats B and B beats C, A must beat C. Human judgments often refuse that order. Rock-paper-scissors is the textbook case; on the same prompt, "more honest" and "more helpful" can also close a loop.
Pairwise classifiers (PairRM / PairPM) concatenate two replies and predict a winner. They can represent any pairwise relation, including cycles, but filling a K-by-K tournament costs O(K²) forwards. Positional encodings and causal attention also make the score depend on which reply is listed first. The quadratic term shows up first when test-time scaling asks a judge to rank dozens of reasoning traces.
This ICML 2025 paper (IIIS Tsinghua, Shanghai Qi Zhi Institute, UCLA; corresponding author Quanquan Gu) asks whether general, possibly intransitive preferences can be modeled with the same linear query cost as BT.
The proposal is preference embedding. Given a prompt x, each reply y is mapped to a vector v in R^{2k}. The preference score is an inner product after a fixed skew-symmetric operator R≻:
s(yi ≻ yj | x) = ⟨R≻ vi, vj⟩
R≻ is block-diagonal, each 2×2 block [[0, -1], [1, 0]], a 90-degree rotation in a plane. The inner product then encodes who sits clockwise of whom: a rock-paper-scissors cycle is three directions spaced around that plane. Two properties follow. s(i≻j) = -s(j≻i), and a reply scored against itself is zero. The operator also preserves vector length, which keeps training numerically stable. When k=1 and v = [r, c], the formula collapses to BT. Theorem 4.4 shows any real skew-symmetric preference matrix can be realized this way, so cycles sit inside the hypothesis class.
The implementation adds two heads on the last hidden state of a language model:
Training is pairwise cross-entropy, same as BT, on the decontaminated Skywork Reward Data Collection (about 80k pairs, two epochs). For alignment they introduce General Preference Optimization (GPO): maximize the log-odds score s, which is unbounded, rather than the win probability P in (0, 1). The iterative update follows the multiplicative-weights pattern used by SPPO, treating a reply's average score against the current policy as a generalized reward. The same score can be plugged into DPO, IPO, PPO, or NLHF.
The sharpest contrast is CyclicPreference, built from Ultrafeedback by keeping prompts with at least three replies whose ratings on instruction following, honesty, truthfulness, and helpfulness form a cycle. Four subsets have 216 to 363 examples. With Gemma-2B-it as the backbone:
| cycle | BT acc. | GPM acc. |
| Honesty ≻ Truthful ≻ Helpful ≻ Honesty | 62.4% | 100% |
| IF ≻ Truthful ≻ Helpful ≻ IF | 61.6% | 100% |
| IF ≻ Honesty ≻ Helpful ≻ IF | 50.0% | 100% |
| IF ≻ Honesty ≻ Truthful ≻ IF | 62.9% | 100% |
BT is a coin flip on the third cycle. GPM is perfect on all four.
On RewardBench, both judges train on the same Skywork data. With Gemma-2B-it, GPM at embedding dim 6 scores 82.29 versus BT's 74.85 (+7.44). Chat moves from 67.32 to 79.61, Chat-Hard from 63.37 to 75.66. On Llama-3.1-8B-Instruct the gap shrinks: GPM dim 8 averages 91.90 versus BT 90.56 (+1.34). Safety and Reasoning do not go GPM's way on the 8B model (Safety 91.08 vs 91.49, Reasoning 95.44 vs 96.47).
Downstream, Llama-3-8B-Instruct is the policy, GPM or BT is the judge, SPPO and GPO each run three iterations. On AlpacaEval 2.0 with GPT-4-turbo, 2B GPM + GPO iter 3 reaches raw win rate 48.25 versus 44.20 for BT + GPO. Length-controlled win rate flips: GPM 37.74, BT 42.21, average length 2582 vs 2151 tokens. The 8B GPM + GPO run stretches to 3249 tokens versus 1969 for BT. MT-Bench is mixed; 8B GPM + GPO iter 3 drops to 7.54, while BT under the same setting sits at 8.26.
A length-normalized GPO variant almost ties LC.WR on the 2B judge (45.55 vs 45.51), with GPM still longer.
Swapping the reward head is cheap: one extra low-dimensional projection, still O(K) queries, no change to Best-of-N or test-time ranking. The gain is largest on a 2B judge and on Chat / Chat-Hard, where "which reply feels better" is the actual question. The perfect cyclic scores make a structural point. If real data contains cross-axis cycles, a scalar reward cannot write them down. Tuning will not invent that missing degree of freedom.
On 8B, RewardBench only moves 1.3 points, and AlpacaEval LC.WR often favors BT. Treat this as an expressivity patch, not a replacement for every reward model. It is most useful when the preference data actually cycles and the judge is small.
The cyclic accuracy is measured on the training set with a different pairing, not a held-out split. Training runs 50 epochs at batch size 1 on 216-363 examples, which is easy to memorize. The cycles themselves are stitched across rating axes, not natural intransitivity under one criterion. Appendix F shows a few raw Ultrafeedback cycles, but the table numbers come from the constructed set.
There is no compute-matched comparison with PairRM, only with BT. Safety does not improve, responses get longer, and MT-Bench regresses on the 8B third iteration. That pattern fits an embedding that rewards covering more facets by writing more. GPO replaces the normalizer log Z with 0; that is reasonable near equilibrium, but the early-iteration bias is not measured. The main text mentions GSM8K and MMLU; the appendix does not report those numbers.