Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs
Kirill Skobelev, Eric Fithian, X. Y. Han
cs.AI
2026-09-15
Collision ratios show SFT is not biased toward mode collapse; about 3,000 survey examples pull four sub-4B models' median R to within 10% of 1.
When LLMs stand in for survey respondents, co-write stories, or brainstorm, the same prompt often yields answers that look more alike than the people they are meant to mimic. That compression is usually labeled mode collapse and treated as a built-in property of the models.
This paper measures the claim. Fix a prompt, draw two independent replies, and record how often they match, or how similar they are under a kernel. Call that collision probability C. The collision ratio R = C(model)/C(target) is 1 when diversity is calibrated, above 1 under collapse, below 1 under over-dispersion. The question is whether ordinary supervised fine-tuning drives R toward 1, or whether the models are structurally stuck on one side.
A bias-variance split of the expected collision gap is the main device. Across independent fits at a fixed context, E[R] − 1 splits into a variance term and a squared-bias term, both nonnegative and both pushing toward collapse, plus a target–bias alignment term that can take either sign. Finite-sample SFT therefore has no built-in direction: a negative alignment term can overpower the other two and leave the model over-dispersed.
A bound controls the size. For any similarity kernel valued in [0, 1], the absolute collision gap is at most the total variation between the product measures, which Pinsker bounds by the square root of KL(target || model). A model close to the population cross-entropy optimum cannot be arbitrarily miscalibrated on diversity.
Three experiments test the split and the bound.
On the synthetic languages a diffuse random-init prior keeps the alignment term negative at every N: small samples over-disperse, most larger N collapse once variance and squared bias dominate. A mode-aligned prior that already concentrates on common tokens flips the alignment term positive at intermediate N, at most 31% of the measured expected gap. Direction is not a model constant. Across 150,000 held-in contexts the ℓ2 bound holds everywhere; the median measured-to-bound ratio is 0.140, the maximum 0.980.
Surveys are sharper. Zero-shot, Qwen3.5-2B sits at median RU 0.99–1.10. The other three models sit at 1.61–2.92, roughly one-third to two-thirds of human effective diversity, worst on the cross-national WVS (gemma-3-4b-it at 2.92). After about 3,000 SFT examples, all twelve survey–model pairs keep median RU within 10% of 1. At the full pool every median rounds to 0.99–1.02.
Small n can move the wrong way. Early GSS SFT lifts Qwen3.5-2B from 0.99 to 2.22 before it recovers.
On CodeNet, comparing the same problems before and after full-data SFT: Gemma-2 1.035 to 0.971, Qwen3.5-2B 0.740 to 0.915, Qwen3.5-4B 0.844 to 1.003, Gemma-3 1.637 to 0.974. Both under- and over-dispersion shrink.
| Setting | Zero-shot median R | After full data |
| GSS / gemma-2-2b-it | 2.11 | 1.00 |
| GSS / Qwen3.5-2B | 0.99 | 0.99 (peaked at 2.22) |
| WVS / gemma-3-4b-it | 2.92 | 1.01 |
| CodeNet / Gemma-3 (same problems) | 1.637 | 0.974 |
| CodeNet / Qwen3.5-2B (same problems) | 0.740 | 0.915 |
For silicon sampling, style matching, or code SFT, "the model is too homogeneous" is often a coverage problem. Enough target-distribution data plus ordinary SFT pulls diversity in from both sides. Specialized diversity losses such as GEM, SED-SFT, or TOFU are not a prerequisite. A few hundred SFT rows can also wreck a model that was already calibrated, which is what happened to Qwen3.5-2B.
The paper does not claim SFT undoes diversity loss from RLHF or DPO. That gap lives in the population optimum of a KL-regularized reward, and it stays as preference data grows. Fixing that still requires changing the objective.
Coverage remains the bottleneck when target data are scarce. The survey target is an unweighted sample that includes train and validation records, not a survey-weighted population; a median R near 1 is not pairwise distributional fit. CodeNet never estimates sequence-level KL, so the square-root bound is not quantitative for the tree-kernel numbers. Qwen3.5-2B defaults to non-thinking output and Qwen3.5-4B to thinking, so cross-model decoding is unmatched. The early-stopped adapter is not guaranteed to be the global validation-loss minimum. Models cap at 4B. Exact sequence collision is nearly unestimable for open-ended text.
Collision is one metric. Form-sensitive and content-sensitive kernels can disagree. Calibration of diversity is not evidence that answers got better.