A General Theoretical Paradigm to Understand Learning from Human Preferences
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, Rémi Munos
cs.AI, cs.LG, stat.ML
2023-10-18
DeepMind casts RLHF and DPO as ΨPO. IPO sets Ψ to identity and regresses the log-ratio gap vs the reference to 1/(2τ), so KL still binds on deterministic preferences.
Alignment data comes as a context, two continuations, and a human choice of which continuation is better. RLHF compresses each pair into a scalar reward with a Bradley-Terry model, an Elo-style score, then maximizes that reward with PPO. A KL penalty is supposed to keep the learned policy near a reference policy πref. Two approximations do the real work. Pairwise preferences are treated as a sigmoid of a reward gap. A reward fit on the logged data is treated as valid on text the policy later samples for itself.
DPO never trains a reward. It updates the policy on the preferences already collected, which removes the second approximation. The loss is still a sigmoid sitting on a log-likelihood ratio. Underneath is the logit, log(q/(1-q)), so Bradley-Terry stays. When preferences get close to win-or-lose, that assumption drains τ. The KL term remains on the page and drops out of the optimum.
Google DeepMind puts these methods under one objective, ΨPO. Ψ is any non-decreasing function of a preference probability. The policy maximizes the expected value of Ψ(p(y beats y′)) and is charged τ times its KL from πref. The completion it is compared against comes from a behavior policy μ.
Set Ψ to the logit and assume the true preferences follow Bradley-Terry. Proposition 1 then says ΨPO, KL-regularized RLHF, and DPO share the same optimal policy. In closed form, π(y) is proportional to πref(y) times exp(τ inverse times the expected Ψ(p)).
The logit has no upper bound. If p(y beats y′) equals 1, Bradley-Terry can represent it only by sending the reward gap to positive infinity. The closed form assigns the loser probability 0 for every finite τ. Finite data makes the same event routine: a true preference of 0.8 can be observed as 1 when few comparisons are in hand, and the empirical optimum again wipes out the loser.
Practical RLHF often avoids that collapse because the reward model underfits. On preferences in {0, 1} the optimal reward is infinite, the network never gets there, and the policy is left nearer πref. DPO skips the reward model and skips that accidental regularizer with it.
IPO sets Ψ to the identity, maximizes total preference, and subtracts the same KL. The optimum is still closed form: π(y) proportional to πref(y) times exp(τ inverse times p(y beats μ)). The offline objective drops the logistic. It turns the optimum into equations and minimizes the squared residual. Theorem 2 says that inside the policies whose support matches μ and πref, the local minimum is unique and is that optimum.
Each logged pair is used in both directions, win and loss. The sample loss reduces to the square of (h minus 1/(2τ)), where h is log(π(yw)/π(yl)) minus log(πref(yw)/πref(yl)). Optimization starts at πref and regresses this gap to the constant 1/(2τ). A larger τ allows a smaller gap. Labels that are entirely 0s and 1s still have a finite target.
No language model is trained, and no human-preference benchmark is reported. Runs on language models are left as follow-up work. The numbers below are context-free bandits.
Two actions, true preference of y1 over y2 equal to 1, uniform πref and μ. DPO returns a policy that selects only y1, for every τ. IPO first gets p(y1 beats μ) = 3/4 and p(y2 beats μ) = 1/4, hence probability σ(0.5/τ) on y1 and σ(-0.5/τ) on y2. As τ goes to infinity the policy returns to the uniform reference. As τ goes to 0 it selects only y1.
Three actions are a softmax, optimized with Adam at learning rate 0.01 and batch size 9 for 18000 steps, 10 seeds, with means and 95% confidence intervals. D1 compares every pair once and yields a total order. In D3, yc never appears, while ya and yb split their comparisons. Across both figures DPO ignores τ, and IPO moves toward πref as τ grows. The plots do not list exact probabilities, and there is no numeric comparison against PPO or SLiC-HF.
| Setting | DPO | IPO |
| Two actions, preference fixed at 1, uniform πref | Winner probability 1 for every τ | Winner probability σ(0.5/τ); uniform again as τ goes to infinity |
| Three-action total order, each pair seen once | Deterministic for every τ tested | Stays non-greedy under strong regularization |
| One action never wins | That probability is 0, for every τ | Larger τ keeps it closer to πref |
On a set of language tasks, DPO matches RLHF, is simpler to implement, and uses fewer resources. The break is narrow and concrete. When a comparison is one-sided, or a completion never wins in the data, τ inside the DPO loss does not control distance to πref. Alignment sets produce both patterns often.
IPO consumes the same records as DPO: context, winner, loser, and reference log-probabilities. Squared error against 1/(2τ) replaces the logistic. Implementation is a change of loss, and a pairwise preference no longer has to be convertible into a pointwise reward.
The fix is regularization when preferences are extreme. There is no new leaderboard. Whether DPO is under-regularized on real preference data, and whether IPO's fixed gap misses graded preferences, is not measured.
Sections 5.3 and 5.4 are labeled as illustrations. The next step named in the text is training language models on human preferences. Three actions, at most three comparisons, and 18000 Adam steps can show DPO ignoring τ. They cannot support a broader ranking of the two losses. The cyclic set D2 is mentioned and not run. SLiC-HF is not placed inside this equivalence.
Every pair is regressed to the same constant, 1/(2τ). Label noise and intransitive preferences are not analyzed. Uniqueness needs μ and πref to share support. Completions absent from the offline data show up only as the D3 sketch, with no sample-complexity bound.
That RLHF survives because the reward underfits is an observation, not a proof. Heavy weight decay on the reward, or empirical preferences that never reach 0 and 1, need not make DPO ignore its KL term. Early stopping is accepted as a further regularizer and is never compared with IPO. Other bounded choices of Ψ, short of the identity, are left open.