LLM creativity lives in the long tail: UChicago study spans 15 benchmarks, 26 open models
Uchicago · hf · 2026-10-14
A University of Chicago / Microsoft / Stanford paper unifies fragmented LLM creativity benchmarks.
- Claim: The shared target of creativity benchmarks can be formalized as task-specific performance in the long tail of a reference input-output distribution, grounded in the novelty-quality product formulation.
- Evidence: Using n-gram absence from pretraining data as a novelty proxy, they evaluate 15 benchmarks across 26 open-weight models. Creativity benchmarks sit further in the tail than standard ones, and rankings on higher-tail subsets align better with human creativity judgments in LM-Arena.
- Output: They introduce q@Tail(α), a low-cost diagnostic measuring mean quality on the tail fraction of an existing benchmark, and show substantial headroom for improving LLM creativity via stronger tail performance.
Related event: Paper Proposes Measuring LLM Creativity via Long-Tail Performance(2 posts)→
More from Research
- DeepSeek's first LLM paper was literally titled "Scaling Open-Source Language Models with Longtermism" — teortaxesTex · 2026-10-15
- Vals AI Puts Frontier Models in Real-Time MuJoCo Robot Simulations — scaling01 · 2026-10-15
- AI Post-Processing Tool Extends Extreme Weather Warnings From 10 Days to 6 Weeks — LesterMackey · 2026-10-15
- DuplexJev Cuts Autoregressive Decoding for Voice Agents, Answering 80 Speech Decisions in 0.1s — adventists-ai · 2026-10-15
- LLM-to-Flow Distillation Adapts Scientific Generators to Natural-Language Discovery Goals — rbhar90 · 2026-10-15
- ETH's Uncertainty-Aware World Model Makes Offline RL Work on Real Robots — breadli428 · 2026-10-15