New paper: LLM outputs are far less diverse than their training data, and temperature tuning won't fix it

rohanpaul_ai · x · 2026-09-04

The paper "Do LLMs Capture the Diversity in their Training Data?" compares model continuations against training continuations for the same prefixes:

Across OLMo, Pythia, and GPT-Neo, training data showed higher conditional diversity under every decoding method tested. For OLMo at 20K samples: training score 338.58 vs 218.88 greedy, 287.31 nucleus, 297.31 ancestral.
Larger models didn't close the gap, and image generators showed the same pattern.
Raising temperature isn't a clean fix: T≈1.9 matched OLMo's training entropy but degraded precision and external conditional NLL.
The authors' alternative generates multiple candidates and reweights plausible but underused outputs, boosting measured diversity without retraining.

Takeaway: LLMs learn a narrower answer distribution than their training data—a structural phenomenon beyond sampling strategy.

More from Research