The Limits of Automatic Evaluation of Creativity in Large Language Models
Alessandro Tutone, Giorgio Franceschelli, Mirco Musolesi
cs.CL, cs.AI, cs.CY
2026-08-25
Blind ratings of 200 short stories: humans barely tell writers apart; Llama-3.3-70B gives AI text perfect 5s, and automatic metrics show near-zero human correlation.
Creative writing is the evaluation task that keeps refusing to sit still. Human ratings are expensive and hard to replicate, so papers reach for two cheaper stand-ins: statistical metrics, and an LLM scoring another model's prose. Both assume the machine score tracks a person. This study tests that assumption on short stories.
A team at Bologna and UCL sampled 100 human stories from Reddit's WritingPrompts, then had GPT-5.2, DeepSeek-V3.2, Mistral Large 3, Claude Sonnet 4.5, and Gemini 3 Pro write 20 stories each. The 200 texts were scored blindly on 11 creativity dimensions by people and by an LLM judge, then compared with Creativity Index, perplexity, EAD n-grams, SBERT-Div, and three syntactic-template scores. The question is blunt: do any of these automatic numbers rank stories the way humans do?
Source labels were stripped for every rater. 115 people produced 441 ratings, 2.21 per text on average; 91.3% were native Italian speakers, and every story was offered in English and Italian. AI stories were written in English then translated. Human stories were translated into Italian with Gemini 3 Pro.
The judge is Llama-3.3-70B-Instruct. To stop one score leaking into the next, each of the 11 dimensions is a separate inference at temperature 0.2, repeated three times. The Likert scale (1 to 5) covers Boden's novelty, surprise, and value, Runco's originality and effectiveness, plus elaboration, fluency, flexibility, authenticity, usefulness, and an overall Creativity score.
Creativity Index uses Infini-gram over RedPajama (about 1.4 trillion tokens) to score 5- to 12-gram L-uniqueness: how much of the text already exists on the web. Perplexity comes from Llama-3-8B Base. Syntactic templates collapse sentences to POS sequences, then report compression ratio (CR-POS), template rate, and templates per token.
Automatic metrics barely move with human scores. Most Spearman ρ values sit in [-0.2, 0.2]. Creativity Index, built to measure creativity, correlates with human Creativity at ρ = 0.07 and never exceeds 0.11 on any of the eleven dimensions. Perplexity is weakly negative: Effectiveness ρ = -0.22, Elaboration ρ = -0.25. People slightly prefer text that is easier for a language model to predict. The strongest positive link is CR-POS with Elaboration, ρ = 0.29: more repetitive syntax reads as more detailed.
| Comparison | Coefficient | Value |
| Creativity Index vs human Creativity | Spearman ρ | 0.07 |
| CR-POS vs human Elaboration | Spearman ρ | 0.29 |
| Llama vs human Elaboration (human stories) | Kendall τ | 0.31 |
| Llama vs human Surprise (human stories) | Kendall τ | -0.01 |
Humans barely separate the two sources. Authenticity 3.27 vs 3.50 (p = .025) and Elaboration 3.17 vs 3.54 (p = .005) are the only significant gaps, both favoring AI. Creativity is 3.16 vs 3.27, p = .569. Llama-as-judge prefers AI on all 11 dimensions, and pins Effectiveness, Elaboration, Fluency, Originality, Value, and Creativity at a perfect 5.00 with zero variance on AI text. Different generators collapse into one ceiling.
Restricted to human-written stories, Kendall τ stays weak. Elaboration 0.31 is the study high-water mark. Creativity is 0.23. Surprise is about zero. The same Llama's perplexity correlates negatively with every score it assigns as a judge. Humans bind Creativity to Novelty, Originality, Surprise, and Value. Llama binds Creativity to Elaboration. The word is being used for two different things.
Any pipeline that lets an LLM grade creative writing, filter training data, or run preference tuning should treat this as a measurement warning. Even under a blind protocol the judge systematically prefers text that looks like itself. Feed that preference back into the data loop and human irregularities become defects, machine style becomes the spec. This is not a leaderboard result. The instrument is picking a side.
The takeaway for practitioners is narrower. Elaboration and fluency still show a faint automatic signal. Surprise, novelty, and overall creativity do not. A human in the loop is still the floor, not a nice-to-have.
The authors already list the main cuts: one task, WritingPrompts only, 200 texts; a single open-weight judge, so GPT or Claude might be milder; default sampling from each vendor, no temperature sweep. One confound they do not flag as hard as it deserves: 91.3% of raters are native Italian, and the Italian of the human stories was produced by Gemini 3 Pro. On that channel the "human" text has already passed through another LLM. 2.21 ratings per story is also thin once you split eleven dimensions.