Harvard study: LLMs judge image creativity zero-shot at 0.68 correlation with humans

DynamicWebPaige · x · 2026-08-15

A new arXiv paper (2606.29672) from Harvard and Penn State asks whether multimodal LLMs can serve as zero-shot judges of visual creativity—scoring originality with no fine-tuning and no examples of human ratings. The poster jokes that after seeing the models' ruthless critiques, she's now scared to ask Gemini or Claude what they think of her art.

Study design

Six multimodal LLMs tested: Gemini 3 Flash, Gemma 4 31B IT, GPT-5.4 Mini, GLM-5v Turbo, Kimi K2.5, and Qwen 3.6 Plus
Datasets: 992 AI-generated images (from human-written prompts) plus 1,500 hand-drawn sketches with human creativity ratings

Key findings

Study 1: All models aligned substantially with human ratings—r = .57–.68 on AI-generated images and r = .29–.68 on sketches
Study 2: Analyzing step-by-step reasoning traces from three models showed that reasoning makes evaluations interpretable (what they attend to, how they balance originality vs. quality, how they justify scores), but did not improve alignment with human ratings

The CoT traces make for fun reading—e.g. "It's not just 'a cone with eyes'; it's a whole scene with a story," or deducting points because "anthropomorphic objects are a known trope in AI art."

Original post →

Harvard study: LLMs judge image creativity zero-shot at 0.68 correlation with humans

More from Research