Structured Output Collapses Answer Diversity Across 44 Language Models
Tapan Parikh
cs.CL, cs.AI, cs.LG
2026-07-21
Re-running the One-Word Census across 44 LLMs, requesting JSON output cuts answer diversity 0.22 bits and shifts 53% of models' default answers; DeepSeek-v3.2 loses half its distinctiveness.
Asking an LLM to pick one answer from a pile of "all correct" options is an arbitrary-looking task that actually exposes a model's personality. Ask 44 models to "name a tree" and their modal answers form a collective word-frequency table; that is the One-Word Census (arXiv:2607.12796). It scores each model against the crowd with surprisal (in bits): rarer answers mean more distinctiveness.
This paper asks a concrete engineering question. If you add a format clause to the prompt, "Reply with JSON only," does the answer the model picks actually change? This is not hypothetical. Nearly every agent, tool-use, and benchmark pipeline defaults to JSON output, and people treat JSON as a neutral shell. The finding here is that it is not neutral at all.
The authors re-ran the One-Word Census: 31 open single-turn prompts (30 category prompts like "name a colour," plus one unconstrained "pick a word"), 44 models, four samples per cell, temperature fixed at 1.0, no system prompt. The only change was a format clause such as "Reply with JSON only, in the form {\"word\": \"<your answer>\"}." No schema enforcement, no constrained decoding, just the request. They then compared surprisal before and after. JSON, XML, YAML, CSV, and brackets were all tested.
Overall, JSON drops the field-mean surprisal from 1.80 to 1.58 bits, a 0.22-bit collapse (p=.0002). On the unconstrained "pick a word" prompt, the modal answer's share rises from 41% to 64%, and the number of distinct answers falls from 52 to 36. Add one line about JSON and the crowd squeezes toward the most common words.
| Model | Surprisal change |
| deepseek-v3.2 | 2.63 to 1.32 bits, half its distinctiveness gone |
| gpt-4o-mini | -0.92 bits |
| gpt-4-turbo | -0.87 bits |
| hermes-4 | -0.91 bits |
| claude-opus-4.8 / sonnet-5 | about 0; already generic, sitting on the floor |
Six of 44 models are significant (BH-FDR q=.10). The Claude models barely move, not because they are immune but because their defaults are already conformist with no room to fall. XML matches JSON (-0.23 bits); YAML and CSV are not significant; brackets slightly reverse the effect (+0.13 bits).
A telling detail: of 144 stable chat defaults, 76 (53%) shift significantly under JSON, and 29% revert outright to the crowd's modal answer. Forcing JSON at the decoder (responseformat) adds only another 0.03 bits on top of the bare request. The authors conclude the convergence lives in the model's response to the register, not in the sampler.
For practitioners this is a side effect worth knowing. If your pipeline depends on output diversity, multi-sample voting, ensembling, or creative generation, a quietly added JSON clause trims that diversity. The same holds for benchmarks: comparing models under different formats measures not only capability but how conformist each becomes when asked for structure. The paper is not against JSON. It is a reminder that JSON is not transparent.
The authors list several: a single OpenRouter snapshot, with temperature 1.0 not uniformly honored; surprisal is panel-relative and not portable across model sets; one clause wording per format; the more common tool-call pathway is untested; and decoder enforcement is heterogeneous across providers. The strong part of this paper is the measurement warning, but it offers no mechanistic explanation of why the register pushes models toward conformity. It documents the effect without explaining it.