Self-Rewarding Language Models
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, Jason Weston
ICML 2024
cs.CL, cs.AI
2024-01-18
Llama 2 70B self-scores with LLM-as-a-Judge; three Iterative DPO rounds lift AlpacaEval 2.0 vs GPT-4 Turbo from 9.94% to 20.44%, above GPT-4 0613.
Standard RLHF trains a reward model on human preferences, then freezes it while the policy trains. DPO skips the reward model and fits the policy on those same human pairs. Both stop where the labels stop. A frozen judge cannot get better while the generator does.
The claim in this Meta and NYU paper is sharper: superhuman agents need superhuman feedback. They put both jobs in one Llama 2 70B. The model writes answers, scores its own candidates with an LLM-as-a-Judge prompt, and trains Iterative DPO on the resulting pairs. The judge is the policy, so scoring can move with generation.
Start from Llama 2 70B. Supervised fine-tune on 3,200 high-quality English first turns from Open Assistant (IFT), plus 1,630 evaluation examples that teach the scoring format (EFT). EFT targets are not human-written rationales. The IFT model scores first; a sample is kept only if its ranking matches human ranks, and over-represented score-4 items are downsampled so the labels do not collapse.
Each iteration then:
Highest versus lowest becomes a preference pair; ties are dropped. M1 yields 3,964 pairs for M2; M2 yields 6,942 pairs for M3. DPO uses β = 0.1.
The additive rubric matters. On the SFT baseline, pairwise accuracy against humans is 65.1% with this prompt and 26.6% with a bucketed multiple-choice prompt from Li et al. Without a usable judge, the loop has nothing to train on.
AlpacaEval 2.0 win rate versus GPT-4 Turbo: 9.94% (M1), 15.38% (M2), 20.44% (M3). M3 sits above Claude 2 (17.19%), Gemini Pro (16.85%), GPT-4 0613 (15.76%), and Llama 2 Chat 70B (13.87%). Most of those systems used large proprietary alignment sets or distillation from stronger models. This run starts from a few thousand Open Assistant seed examples.
GPT-4 pairwise on 256 prompts: M2 beats M1 55.5% to 11.7%; M3 beats M2 47.7% to 12.5%; M3 beats the SFT baseline 62.5% to 9.8%. Three authors, blind, on 50 prompts: M3 versus SFT is 66% to 18%. Same direction.
| Model | AlpacaEval 2.0 | MT-Bench | Judge pairwise acc. |
| SFT (IFT only) | n/a | 6.85 | 65.1% |
| M1 (IFT+EFT) | 9.94% | 6.78 | 78.7% |
| M2 | 15.38% | 7.01 | 80.4% |
| M3 | 20.44% | 7.25 | 81.7% |
MT-Bench gains concentrate in writing, roleplay, extraction, and STEM. Math and code barely move. Adding 11,254 perfect-score answers as extra SFT data does not help (29% versus 30% against the baseline). Starting without EFT collapses scores toward 4, leaving only 541 then 429 valid pairs, and those models lag the IFT+EFT line.
The judge itself improves with no extra human scoring labels: pairwise accuracy climbs from 78.7% (M1) to 81.7% (M3). The paper's account is transfer: better instruction following makes the judging instruction better too.
The transferable piece is the loop: one set of weights as policy and judge, Iterative DPO moving both. Constitutional AI and RLAIF use a separate LLM to write feedback, then train a frozen reward model. Here the judge is the policy, so later rounds can in principle get cleaner pairs from a stronger scorer.
The seed data is chat and writing. Math, code, and logic barely benefit. Anyone who wants this for hard reasoning has to change the seed tasks. Open Assistant prompts will not grow a solver on their own.
Only three iterations, one 70B setting. Saturation is unknown. Mean AlpacaEval generation length grows from 1,092 (M1) to 1,552 (M2) to 2,552 (M3). Length correlates with judged quality; the paper flags this and does not isolate it.
Training rewards and several evals are all LLMs: self-judge at train time, GPT-4 on AlpacaEval and MT-Bench. Human ratings agree in direction, but 50 author-labeled prompts do not rule out reward hacking. There is no safety eval. New prompts in the main run still come from frozen Llama 2-Chat, not live users.
Read this as "after RLAIF, fold the judge into the same iterating policy." It does not show superhuman feedback. It shows that a self-scoring DPO loop can run three rounds and climb on both axes under these evals.