How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
Ferenc Huszár
stat.ML, cs.AI, cs.IT, cs.LG
2015-11-17
Reframes scheduled sampling as a KL divergence and proves it inconsistent, then argues generative models should target reverse KL, which is why GANs yield more realistic samples.
Around 2015, autoregressive models (RNNs and LSTMs that predict one symbol at a time) were the default for generating text and image captions. They were trained by maximum likelihood (ML): maximize the probability of predicting each real token given the real prefix. Generation, however, broke down.
At training time the model only ever sees prefixes from real data, so every prediction sits on clean input. At generation time it has to feed on its own past output. Sample one slightly wrong token and the next prediction conditions on that error; mistakes compound and the sequence degenerates into gibberish. Bengio et al. (2015) named this the train-generation mismatch and offered a popular fix: scheduled sampling, which during training randomly swaps the real input for a sample drawn from the model itself, so the model gets used to its own output. It powered the winning entry of the 2015 MSCOCO image captioning benchmark.
Huszár asks two deeper questions. Did scheduled sampling actually fix anything? And is maximum likelihood the right objective for a generative model in the first place?
The first cut lands on scheduled sampling. Huszár rewrites its training objective as a combination of two KL divergences (a non-symmetric measure of distance between distributions) and works it out for the simplest case of length-2 sequences. Maximum likelihood is equivalent to minimizing the forward KL, KL[P||Q], where P is the real data distribution and Q the model. When scheduled sampling replaces the first token with the model's own sample, the objective becomes a second term in which the second-token prediction is no longer conditioned on the real first token. The full objective is a convex combination of the two, governed by epsilon (annealed from 1 down to 0).
The statistical yardstick for a good objective is the strictly proper scoring rule: with a flexible enough model class and enough data, the minimum must sit at Q=P. That property guarantees consistency. After rearranging, Huszár shows that as epsilon goes to 0 the divergence is minimized not at the true joint P but at the factorized distribution P(x1)P(x2). The harder you train, the more the optimum says "ignore the prefix, just memorize the per-position marginal." In RNN terms, the optimal architecture uses its hidden state as a position counter and ignores the prefix content entirely. Intermediate values of epsilon are inconsistent too; the optimum sits between P and the factorized distribution.
That is why scheduled sampling "works" without fixing the root cause. It does not address maximum likelihood's inherent flaw; it nudges the model toward a trivial solution that recovers from mistakes by giving up on context. As for the MSCOCO win, Huszár guesses it was a side effect of early stopping, random restarts, model structure and the annealing schedule colliding, since optimization never actually reached that trivial solution.
This is a theory paper with no benchmark experiments. The "results" are a few mathematical claims and one illustration.
The headline: scheduled sampling is provably inconsistent in the length-2 case; it cannot recover the true distribution. For training objectives, the paper lays out a single spine:
| Objective | Equivalent KL | Behavior | Effect on samples | ||
| Maximum likelihood | KL[P | Q] (forward) | Covers all modes, spills mass where P has none | Over-generalizes, emits implausible samples | |
| Reverse KL (ideal) | KL[Q | P] | Locks onto the largest mode, puts no mass where P has none | No absurd samples, at the cost of dropping minor modes | |
| Generalized JS, JSpi | pi->0 like ML, pi->1 like reverse KL | Interpolates | Tuning pi trades coverage for mode-seeking |
A 2D Gaussian demonstration (Figure 1) shows it: P is an anisotropic multi-modal Gaussian approximated by a single isotropic Q. At pi=0.1, Q spreads to cover all modes (like ML); at pi=0.5 it is the standard JS that GANs approximately minimize; at pi=0.99 it collapses onto the dominant mode (like reverse KL). The key proof: as pi->0, JSpi/pi tends to KL[P||Q]; as pi->1, JS{1-pi}/(1-pi) tends to KL[Q||P]. So pi is a continuous dial between maximum likelihood and the ideal reverse KL.
The parameter pi has a concrete operational meaning: the real-to-fake sample ratio the discriminator sees. pi=0.5 is the balanced classification of standard GAN; pi<0.5 over-samples real data; pi>0.5 biases toward generated data. Turning that one ratio is equivalent to shifting between "cover every possibility" and "emit only the safest samples."
The paper resolves two things people could not articulate. First, scheduled sampling is not a free lunch: it "fixes" exposure bias by pushing the model toward a context-blind trivial solution. Anyone doing sequence generation still hits this wall; the paper offers a clean theoretical warning rather than yet another schedule. Second, it gives an information-theoretic account of why GAN samples look sharper than VAE or ML samples. GANs approximately minimize JS, which leans toward the mode-seeking behavior of reverse KL, so it tolerates dropping coverage to avoid implausible samples. VAEs maximize the evidence lower bound (ELBO), placing them in the maximum-likelihood / forward-KL family, so they cover everything and blur. The later practice of using VAEs for diversity, GANs for sharpness, and hybrids of the two traces back to these two KL directions.
For today's practitioners the direct payoff is limited. These objectives are no longer hand-tuned in the era of large models. But the single line "forward KL covers, reverse KL seeks" still explains a string of phenomena: why autoregressive RNN sampling avalanches, why GAN samples are conservative, why diffusion models sidestepped the whole debate. It is the foundation under a large chunk of the generative-modeling literature that followed.
The author names several. Adversarial training was young: it depends on sampling from Q, which is prohibitively inefficient in high dimensions, so it mostly worked on low-dimensional problems and was sensitive to hyperparameters; how to apply it to discrete generation (text, where sampling is not differentiable) was unresolved. His own pi-tuning modification may hurt convergence, and the paper explicitly says it was not studied.
A few more caveats. The inconsistency proof holds rigorously only for length-2 sequences; Huszár calls this an idealized form, and the extension to longer sequences is intuition, not theorem. "GANs minimize JS divergence" is itself an approximation: the original GAN uses a non-saturating loss, and exact JS minimization requires an optimal discriminator, a gap the paper glides over. Reverse KL as the "ideal perceptual objective" rests on a strong assumption, that the human observer has learned an accurate P. That reads more like an analogy than a verifiable model. And there is no experiment showing that tuning pi actually buys better samples; the practical value of generalized JS stays at the level of the illustration.