What creativity leaderboards actually measure: 15 benchmarks point to long-tail performance
shangbinfeng · x · 2026-10-15
A new paper from University of Chicago, Stanford and Microsoft researchers argues that LLM creativity benchmarks, despite differing in task and metric, effectively measure one shared target: long-tail performance against a reference input-output distribution.
- The study spans 15 creativity benchmarks and 26 open-weight models, using n-gram absence from pretraining data as a novelty proxy
- Findings: creativity benchmarks sit further into the long-tail regions than standard benchmarks
- Across both benchmark types, model rankings derived from high-tail subsets align more closely with human creativity judgments in LM Arena
- The authors introduce q@Tail0, a low-cost diagnostic measuring mean task quality on the tail fraction of existing prompts
The upshot: when a new model climbs a creative-writing leaderboard, what improved is largely its tail performance.
Related event: Paper Proposes Measuring LLM Creativity via Long-Tail Performance(2 posts)→
More from Research
- Curated roundup: frontier models help crack decades-old problems across disciplines — IgorCarron · 2026-10-15
- Stanford's CausalSmith automates causal inference research with Lean-verified proofs — StanfordAILab · 2026-10-15
- Cellular Intelligence wins Best Poster at scverse 2026 for perturbation platform — arjunrajlab · 2026-10-15
- BARRACUDA microfluidic chip traps a million single cells to map gene circuit design space at scale — NikoMcCarty · 2026-10-15
- Stanford's DexAgent turns one human egocentric video into dexterous robot training data — StanfordAILab · 2026-10-15
- Johns Hopkins wins HCOMP 2026 Best Paper for reader-centered summarization evaluation — mdredze · 2026-10-15