Deep reinforcement learning from human preferences
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei
stat.ML, cs.AI, cs.HC, cs.LG
2017-06-13
By learning a reward model from human comparisons, this OpenAI-DeepMind team matched real-reward RL on Atari and robotics with under 1% of interactions and an hour of feedback.
By 2017, deep reinforcement learning could play Atari and beat the world Go champion, but only in domains that handed the agent a well-defined reward function. Plenty of real tasks do not. If you want a robot to wipe a table or scramble an egg, the reward has to be a function of the robot's sensors, and any simple version is easy to game. The agent maximizes the score you wrote, not the behavior you wanted.
This is an early form of the alignment problem: the agent optimizes the objective you gave it, and that objective does not actually capture your intent.
The usual alternatives all fall short. Imitation learning and inverse reinforcement learning need human demonstrations, but some behaviors are impossible for a human to demonstrate, like a backflip on a non-humanoid robot. Using raw human feedback as the reward is too expensive; deep RL routinely needs hundreds or thousands of hours of experience, so paying a person to score every step is hopeless. To make human feedback practical, it has to be cut down by several orders of magnitude.
The core move is to split learning the goal into two networks trained separately: a policy that acts in the environment, and a reward model that predicts how good a clip of behavior is.
Training runs as three asynchronous loops:
The key design choice is to ask humans for comparisons rather than absolute scores. Comparing two one-to-two-second clips is much easier than scoring them, especially in continuous control, where the scale of an absolute score drifts and humans cannot give it consistently.
The reward model is fit with the Bradley-Terry model: treat as a latent factor, and let the probability of preferring one clip rise exponentially with its summed reward. This is the Elo rating system from chess in disguise. The difference in predicted reward between two clips maps to the probability a human picks one over the other, just as an Elo gap maps to win probability. The model is fit by cross-entropy.
A few engineering choices hold it together:
For the policy, the paper uses A2C on Atari and TRPO on robotics, both standard at the time. Predicted rewards are normalized to zero mean and fixed variance, since the absolute level of the reward is not determined by the problem.
Feedback covers under 1% of the agent's interactions with the environment, and the interaction complexity drops by roughly three orders of magnitude.
On the robotics tasks (eight MuJoCo tasks including hopper, walker, half-cheetah, and ant), 700 human comparisons roughly match RL trained on the true reward, and 1,400 synthetic labels edge past the true reward, likely because the learned reward is better shaped. Real human feedback lands anywhere from half as efficient to on par with the oracle. On ant, human feedback clearly beats synthetic labels, because the authors asked humans to prefer clips where the robot stays upright, which turns out to be a useful shaping signal.
Atari (seven games) is harder to match. BeamRider and Pong approach or match RL with only 3,300 synthetic labels; Seaquest and Qbert catch up but learn slowly; SpaceInvaders and Breakout never match RL, though the agent still improves, reaching about 20 on Breakout and up to 50 with enough labels. With real human feedback, Qbert fails to clear the first level.
The clearest demonstration is a set of novel behaviors that have no ready reward function:
| Behavior | Human queries | Scale |
| Hopper backflips | 900 | under 1 hour |
| Half-cheetah on one leg | 800 | under 1 hour |
| Enduro pacing other cars | 1,300 | 4M frames |
On cost, contractors give each comparison in 3 to 5 seconds, and a single task takes 30 minutes to 5 hours of human time. Atari training costs about $25 in compute; 5,000 labels correspond to about 5 hours of labor, roughly $36 at US minimum wage. Human time and compute are already on the same order.
This is the seed of RLHF. The reward-model-plus-policy loop, the reward model trained on pairwise comparisons, and the online collection of preferences that ChatGPT and InstructGPT use all come straight from here; the environment just changed from Atari and robots to language models.
The practical takeaway: complex goals do not always need a hand-coded reward function; you can teach them with preference comparisons. Comparisons are easier to label consistently than scores. But this paper is about control in simulators, at a scale nothing like modern language-model RLHF. It is the seed, not the finished product.
The authors are deliberately modest, repeating that the goal is to get close to true-reward RL, not to beat it. Real human feedback at the same label count is usually a bit worse than synthetic labels, often comparable to synthetic labels with 40% fewer of them. The causes are human error, inconsistency between contractors, and uneven labeling rates that concentrate labels in narrow parts of the state space.
The active clip selection (querying the pairs where the ensemble disagrees most) is something the authors admit hurts on some tasks; it is a crude approximation. Training the reward model offline breaks down: the distribution of states the agent visits drifts as it learns, so an offline reward model captures only part of the true reward, and maximizing that partial reward sends the agent off course. On Pong, offline training sometimes makes the agent avoid losing without ever scoring, producing rallies that never end. Human feedback has to be woven into RL, not handed over once.
The authors also do a lot of manual surgery to stop the environment from leaking the task: they remove episode termination and life-loss signals and blank out the score display. That engineering effort itself signals how fragile learning purely from human preferences still is.