Scaling Laws for Reward Model Overoptimization
Leo Gao, John Schulman, Jacob Hilton
cs.LG, stat.ML
2022-10-20
Using a fixed gold reward model as ground truth, this work measures how gold reward first rises then falls as you optimize a proxy reward model, and fits scaling laws for the coefficients, quantifying Goodhart's law in RLHF for the first time.
In RLHF everyone optimizes a learned reward model that acts as a proxy for human preference. Goodhart's law says that when a measure becomes a target it stops being a good measure, so optimizing a reward model too hard should eventually hurt true (gold) performance. The effect was widely observed but never carefully measured, because collecting human preference data is expensive. This paper measures how large overoptimization is and how it scales.
To bypass data cost the authors use a synthetic setup: the 6B InstructGPT reward model plays the gold-standard human, generating 100,000 preference pairs used to train proxy reward models from 3M to 3B parameters. They optimize the proxy with two methods, best-of-n sampling (BoN) and PPO reinforcement learning, and track gold score against the square root of KL divergence from the initial policy, written d.
The result is two clean scaling laws. BoN gold score fits R(d)=d(α−β·d), and PPO fits R(d)=d(α−β·log d). Both rise then fall, and the fall is overoptimization. The coefficients α and β vary smoothly with proxy parameter count (roughly logarithmic), with PPO's α nearly independent of model size. Given a scale you can predict how high gold score will reach.
Several findings stand out. First, a data threshold: below about 2,000 comparisons the reward model is near chance regardless of parameters; past it, more data means less goodharting. Second, policy size is counterintuitive: larger policies gain less from optimization but do not overoptimize faster, peaking at almost the same KL with almost the same proxy-gold gap. Third, BoN and PPO spend KL very differently, PPO far less KL-efficient, so KL is a poor cross-method metric for optimization; but plotted against proxy score the two look similar. Fourth, a KL penalty here acts only like early stopping, making gold converge earlier without improving the gold-KL frontier.
The authors map this onto a Goodhart taxonomy: the α term is regressional Goodhart (the proxy depends on noisy features), and the β term is extremal Goodhart (optimization pushes the distribution outside the reward model's training support). Iterated RLHF, repeatedly retraining the reward model on fresh human feedback, helps the β term by an amount proportional to d·log(k) but does not help the α term.
This is the foundational work that turns "do not over-optimize the reward model" from intuition into a predictable engineering quantity. For RLHF practitioners it gives concrete levers: make the reward model large enough, collect at least a few thousand preferences, treat the KL penalty as roughly early stopping rather than a savior, and use online iteration with fresh feedback to actually reduce overoptimization. The smooth coefficient scaling means you can fit on small scale and extrapolate to predict large-scale behavior.
The synthetic setup may not transfer: the gold model is itself a model, and the gap between true human intent and gold labels is uncaptured, which the authors call their biggest limitation. All experiments use the InstructGPT environment, so generalization is unverified. The β term predicts unbounded utility loss in the limit, but real systems hit other constraints first. The models are not yet powerful enough to produce adversarial Goodhart, likely the most dangerous form in the future, at which point these scaling laws may break entirely.