Evaluation of Text Generation: A Survey
Asli Celikyilmaz, Elizabeth Clark, Jianfeng Gao
cs.CL, cs.LG
2020-06-26
A Facebook AI, UW and Microsoft survey sorts NLG evaluation into three families and shows why BLEU-style scores collapse on open-ended generation tasks.
Text generation models kept getting better while the instruments for judging them stayed frozen in the machine translation era. By the time GPT-2/3 could produce long texts that read as human-written, the standard tools were still n-gram matching formulas designed for tasks with a single reference answer. Human evaluation is the nominal gold standard, yet the survey documents how poorly it is run: when Lee et al. sampled NLG papers from ACL and INLG, only 57% reported how many people judged the outputs, and in a 135-paper sample of generation papers only 18% reported any inter-annotator agreement. Researchers from Facebook AI, the University of Washington, and Microsoft Research wrote a survey dedicated entirely to NLG evaluation, grouping methods into three families and closing with case studies of document summarization and long text generation.
The taxonomy is cut along two axes: how much the metric depends on humans, and whether it requires training.
The thread holding the survey together is an assumption check. Untrained metrics presuppose substantial word overlap between output and reference. Open-ended tasks such as chit-chat and story generation admit many valid answers, so the assumption collapses.
The most convincing evidence sits in the survey's Table 3. In a dialog example, the model reply "Oh sure! Heard the film about Turing is out!" is perfectly plausible in context yet scores 0.0 on BLEU, ROUGE, and WMD against the reference; a near-paraphrase image caption reaches only 0.20 BLEU.
| Finding | Source | Number |
| Hallucinated single-sentence summaries | Maynez et al. (2020) | >70% |
| ACL 2018 generation papers with human eval | Hashimoto et al. (2019) | 20 / 26 |
| Papers reporting judge counts | Lee et al. (2021) | 57% |
| Papers reporting inter-annotator agreement | Amidei et al. (135-paper sample) | 18% |
| Papers not stating evaluation dimensions | Howcroft et al. (20-year sample) | 25% |
Verdicts compiled across the literature: BLEU is unvalidated outside machine translation (Reiter, 2018), and its correlation with human judgments drops noticeably once outliers are removed (Mathur et al., 2020). Caccia et al. found that text with perfect BLEU is grammatical yet lacking in semantic and global coherence. On summarization, SMD correlates with human judgments better than ROUGE; on image captioning, SPICE beats CIDEr.
This is a map for metric selection. Before benchmarking a generation system, ask whether the task has a single correct output: if yes, untrained metrics are cheap and stable enough; if no, BLEU-style scores can assign zero to perfectly reasonable outputs. The machine-learned branch, ADEM and HUSE learning scorers from human judgment data, is the direct ancestor of today's LLM-as-judge; the paradigm was fully laid out in 2020, and HUSE even catches low-diversity outputs that human judges miss. The evaluation debates raging again in the LLM era, metric-human mismatch, hallucination, judge reliability, were all documented here with evidence. It proposes no new method; the value is entirely in the compilation.
Literature coverage ends in the GPT-3 era (v2 updated May 2021): no instruction-following models, no LLM-as-judge. The authors themselves warn that standardized evaluation platforms bring consistency together with rigidity and the risk of overfitting the wrong objectives. ADEM-style judges are documented as conservative, assigning low scores to plausible responses, and learned metrics inherit the biases of their training data. One practical caveat: every number in the tables comes from a different paper with different setups, so they are not directly comparable; the survey compiles evidence rather than running controlled comparisons.