Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica
NeurIPS 2023 Datasets and Bench
cs.CL, cs.AI
2023-06-09
The paper behind MT-Bench and Chatbot Arena validates LLM-as-a-judge: GPT-4 agrees with human experts at 85%, above the human-human rate, with measured and mostly fixable biases.
Chat models tuned with instruction data and RLHF are the ones users prefer, and classic benchmarks barely register the difference. MMLU and HELM test closed-ended knowledge, where an unaligned base model like LLaMA-13B posts respectable scores; on open multi-turn dialogue it falls apart, repeating itself where Vicuna-13B, fine-tuned from that same base, walks through interest rates, inflation, and jobs. Benchmark scores and user perception have come apart. Human evaluation is the gold standard and is slow and expensive, so the paper builds a scalable automated substitute: a strong LLM as judge, tested against that gold standard.
Two benchmarks with human preference labels come first.
Three judge modes follow: pairwise comparison, single-answer grading, and reference-guided grading. Pairwise resolves fine differences but the number of pairs grows quadratically; single-answer grading scales but its absolute scores drift when the judge model changes.
The core contribution is measuring the judge's biases one by one. Position bias: on near-identical answers judged twice with swapped order, Claude-v1 is consistent only 23.8% of the time, GPT-3.5 46.2%, GPT-4 65.0%. Verbosity bias: padding a list answer with a rephrased copy of itself fools Claude-v1 and GPT-3.5 91.3% of the time, GPT-4 8.7%. Math grading fails in a stranger way: GPT-4 solves a problem correctly on its own, then gets misled by a wrong candidate answer placed in context.
The fixes are plain engineering. Judge twice with positions swapped and count a win only if it holds both times, otherwise call it a tie. Few-shot examples push GPT-4's consistency from 65.0% to 77.5% at 4x the API cost, so they stay off by default. For math, the judge solves the problem first and grades against its own reference, cutting failures from 14/20 to 3/20.
| Comparison | Metric | Result |
| GPT-4 pairwise vs human experts (MT-bench, non-tie) | Agreement | 85% |
| Human experts vs each other (same setup) | Agreement | 81% |
| GPT-4 pairwise vs crowd users (Arena, non-tie) | Agreement | 87% |
| Claude-v1 / GPT-3.5 / GPT-4 | Swap consistency | 23.8% / 46.2% / 65.0% |
| Math grading (default / CoT / reference) | Failure rate | 14/20 / 6/20 / 3/20 |
GPT-4 agreeing with human experts more often than humans agree with each other is the load-bearing number. Agreement also climbs from 70% to nearly 100% as the win-rate gap between the two models widens; close matchups are where judges wobble. Shown GPT-4's reasoning after a disagreement, humans called it reasonable 75% of the time and changed their vote 34% of the time.
The benchmark separates models that MMLU compresses together: on MT-bench's 10-point scale, GPT-4 scores 8.99, GPT-3.5 7.94, Vicuna-13B 6.39, LLaMA-13B 2.61, against MMLU scores of 86.4, 70.0, 52.1, and 47.0. Vicuna-7B tuned on just 4.8M tokens of curated conversations jumps from 2.74 to 5.95 on MT-bench while MMLU edges from 35.2 to 37.3. A small dose of good dialogue data teaches style, not knowledge, so the two kinds of benchmarks need to sit side by side.
This is the paper that turned LLM-as-a-judge from a convenient trick into an evaluation paradigm with evidence behind it. MT-bench scores became a staple of model cards, the Arena leaderboard became the public battleground for model rankings, and the released 3K expert votes plus 30K conversations became base material for later judge research. Three rules carry straight into practice: use GPT-4 single-answer grading for fast, stable evals; always swap positions in pairwise mode; supply references for math and reasoning questions.
To be honest, having an LLM grade answers was not new at the time, and the paper says as much about its own earlier blog post and concurrent work. What it added was the first systematic check against a human gold standard, with every bias measured and put on the table.
The authors name three: only helpfulness is measured, leaving honesty and harmlessness untouched; accuracy, relevance, and creativity collapse into a single score; the bias mitigations are preliminary. The origin of position bias, training data or the left-to-right causal transformer, stays a guess.
A few more things deserve caution. The human gold standard is 58 graduate-level labelers and about 3K votes, a small pool. The 75% "reasonable" figure was collected after GPT-4's verdict was shown, which invites anchoring. The Arena agreement check rests on a 3K sample of single-turn votes. The paper concedes it cannot settle self-enhancement bias, yet GPT-4 giving itself a 10% higher win rate and Claude-v1 25% is exactly the pattern to watch whenever judge and contestant share a vendor. And every headline result rests on GPT-4, the strongest judge available at the time; one tier down, swap-consistency collapses to 23.8%, and the paper has no data on where the usable-judge boundary lies.