Holistic Evaluation of Language Models
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, Yuta Koreeda
cs.CL, cs.AI, cs.LG
2022-11-17
Stanford CRFM evaluates 30 language models on 16 core scenarios and 7 metric families, raising scenario coverage from 17.9% to 96.0% and reporting 25 cross-cutting findings.
By 2022 language models were already generic text interfaces, while evaluation was still a private hobby of each paper. T5 and Anthropic-LM did not share a single dataset in their original writeups. Several commercial models had no public scores at all. On HellaSwag, one paper reported fine-tuning, another 0-shot, another 5-shot. Calibration, robustness, fairness, and toxicity, when measured at all, lived on separate datasets, so trade-offs at the point of use stayed invisible.
Stanford CRFM's Holistic Evaluation of Language Models (HELM) evaluates the language model, not a task-specific system. It first writes down a taxonomy of scenarios and metrics, then picks a feasible subset and states what the taxonomy still lacks. The same 5-shot prompts and the same core scenarios put 30 open, limited-access, and closed models on one comparable sheet.
A scenario is a (task, domain, language) triple. The core set has 16 user-facing scenarios: question answering, retrieval, summarization, sentiment, toxicity detection, text classification, and related tasks. The language is English, including African American English and national varieties. Each core scenario is scored on seven metric families when possible: accuracy, calibration, robustness, fairness, social bias, toxicity, and efficiency. Of 112 (scenario, metric) cells, 98 are filled, 87.5%.
Seven targeted evaluations add 26 scenarios for linguistic skill, knowledge, reasoning, copyright regurgitation, disinformation, bias, and toxicity elicitation. Twenty-one of the 42 scenarios had not been used in mainstream LM evaluation.
Adaptation is plain 5-shot prompting, without chain-of-thought or prompt tuning, on purpose: the object is a generic text interface, not each model's private ceiling. Thirty models from 12 organizations are included, from GPT-J 6B to TNLG v2 530B: 10 open, 17 limited-access, 3 closed. The run is 4,939 evaluations, about 12.2 billion tokens and 17.4 million queries, $38,001 on commercial APIs, and roughly 19,500 GPU hours for open models. Prompts, completions, and a modular toolkit were released.
Before HELM, models had been evaluated on 17.9% of the core scenarios on average, and some pairs of prominent models had no scenario in common. HELM raises that to 96.0% and reports 25 cross-cutting findings. A few that can be checked:
| Finding | Number | Comparison |
| Core-scenario coverage | 96.0% | 17.9% before |
| Instruction tuning | text-davinci-002 leads accuracy, robustness, and fairness | Anthropic-LM 52B is top-3 on all three, an order of magnitude smaller than TNLG 530B |
| GSM8K | code-davinci-002 52.1% | text-davinci-002 35.0%; no other model above 16% |
| TruthfulQA | text-davinci-002 62.0% | next is Anthropic-LM at 36.2% |
| NarrativeQA | text-davinci-002 74.4% ROUGE-L | UnifiedQA-v2 at 67.4%; TNLG falls from 72.6% to 38.9% under robustness perturbations |
| HellaSwag format | OPT-175B at 79.1% with separate 0-shot choices | 30.2% when choices are packed into one 5-shot multiple-choice prompt |
| BBQ | text-davinci-002 89.5% | T0++ at 48.4%; the three most accurate models reverse the bias direction of the rest in ambiguous contexts |
On this snapshot, instruction tuning beats raw scale to 530B. Open models lagged closed ones across core scenarios. Accuracy, robustness, and fairness move together, but the most accurate model is not always the most robust. Average toxicity and bias on core scenarios are low; targeted toxic prompts still raise toxic completions. Bits-per-byte on The Pile does not predict downstream accuracy across families. Inside a family, scale predicts accuracy; across families it does not. Every model that wins head-to-head accuracy comparisons well above chance is at least 50B parameters.
HELM changes the unit of evaluation from a list of datasets to a scenario-by-metric matrix, and it forces non-accuracy metrics onto the same sheet as accuracy. For someone picking a model now, the 2022-2023 ranking is stale. The method is not: same prompts, same scenarios, many metrics, and an explicit list of holes. Later HELM releases, HELM-lite, instruction-model evals, and multimodal follow-ups all grow this taxonomy.
The study also measures the evaluation itself. Prompt formatting, in-context examples, and whether multiple-choice options are joined or split can move one model from near chance to near the then-best score. Standardizing the protocol and letting each model use its favorite prompt are in real conflict.
The authors split the caveats in three. On results: 5-shot only, so fine-tuning or optimized prompts may change the trends; pretraining contamination cannot be ruled out; decoding hyperparameters were barely swept. On implementation: core scenarios are English and classical user tasks, with little biomedical, financial, educational, or non-English coverage; toxicity uses PerspectiveAPI; about 1,000 instances and 3 seeds per scenario, so some comparisons may not be significant. On design: the output is a matrix, with no official scalar score and no total order.
For readers now, the table stops at the davinci / OPT / BLOOM generation. It has no GPT-4-class chat models and no long-context setting. Automatic metrics on CNN/DailyMail and XSUM already failed to match quality differences seen by annotators. As a panorama of that moment, the material is complete. As a buying guide for today, later HELM versions are the ones to read.