Generating Sequences With Recurrent Neural Networks
Alex Graves
cs.NE, cs.CL
2013-08-05
An LSTM that predicts one point at a time and feeds its own samples back as input generated coherent multi-page Wikipedia (1.33 BPC) and handwriting that passed blind taste tests.
In 2013 sequence generation was still mostly counting. n-gram models counted how often recent character combinations appeared in the training set and guessed the next symbol from those counts; compressors like PPM did the same thing under the hood. Their shared weakness was exact matching: if a recent context never appeared verbatim, the model was stuck, and the approach did not scale. RNNs could in principle learn fuzzy, interpolative predictions, but ordinary RNNs forgot the distant past, gradients vanishing as they propagated, so left to generate on their own they drifted and fell apart.
Graves set out to do two things. First, make an RNN actually generate complex sequences with long-range structure that stays coherent over a thousand steps, not just predict. Second, a harder concrete problem: handwriting synthesis. Given a sentence, write it in cursive. The catch is that the alignment between characters and pen strokes is unknown. One writer finishes a letter in three strokes, another in one continuous sweep, and the model must learn by itself which character it is currently writing.
The core idea is one sentence: predict the next data point, then feed the model's own sampled prediction back as the next input. During training the network processes real data one step at a time and predicts a distribution over the next input; during generation it samples from that distribution, feeds the sample back in, and repeats. Graves likens it to dreaming, the network treating its inventions as real. That is autoregressive generation.
The backbone is a stack of LSTM layers with skip connections from the inputs to every hidden layer and from every hidden layer to the outputs, shortening the gradient path and taming vanishing gradients. The output distribution is matched to the data type: discrete text uses a softmax over a character-level multinomial, characters rather than words for maximum generative flexibility; continuous handwriting coordinates use a mixture density network, a mixture of bivariate Gaussians for the pen's (x, y) offset plus a Bernoulli for whether the pen lifts. The number of mixture components is how many distinct moves the network thinks the next point could be.
The key novelty for synthesis is the soft window, an alignment mechanism. A text is a sequence of characters and the trace is a sequence of coordinates, very different lengths with no pre-existing alignment. Graves slides a mixture of Gaussians across the character sequence; at each step the network outputs the window's location kappa, width beta, and weight alpha, deciding which character to focus on. One design choice matters: the location is defined as an offset from the previous location, constrained to be positive, so the network learns how far to slide rather than an absolute position. The paper states plainly that without offsets the network never learns to align.
Generation adds two control knobs that look familiar today. Biased sampling deliberately shrinks the variance of the Gaussian components so samples favor higher-probability strokes, yielding neater, more legible handwriting; pushed to the limit it collapses into extremely regular "average handwriting." That is sampling temperature. Primed sampling first feeds a real writer's strokes to lock in a style, then continues generating in that style. The prototypes of temperature control and conditional generation are both here.
On Penn Treebank, with adaptive weight noise and dynamic evaluation, the character-level network reached 1.24 BPC (122 perplexity) and the word-level network 1.23 BPC (117 perplexity). For comparison, Mikolov's thesis records 141 perplexity for a 5-gram with Kneser-Ney smoothing, 131.1 for PAQ8 compression, and 123.2 for a dynamically evaluated word-level RNN. The author concedes the word-versus-character comparison is "somewhat unfair" because the word-level network had far more parameters, and an ensemble of multiple RNNs, a 5-gram, and a cache model reached 89.4, beyond a single model's reach.
| Task | Setup | Result | Baseline |
| Penn Treebank (char) | adaptive wt. noise + dynamic eval | 1.24 BPC / 122 ppl | 5-gram KN 141 |
| Penn Treebank (word) | same | 1.23 BPC / 117 ppl | dynamic word-RNN 123.2 |
| Wikipedia (Hutter) | 7-layer LSTM + dynamic eval | 1.33 BPC | PAQ-8 variant 1.28 (with code) |
| Handwriting synthesis | adaptive wt. noise | log-loss -1128.2 nats, SSE 0.23 | prediction net SSE 0.41, down 44% |
Wikipedia is more striking. On the 100-million-byte Hutter Prize dataset, a seven-layer LSTM of 700 cells each (about 21.3M weights) reached 1.33 BPC with dynamic evaluation (1.67 static). The then Hutter Prize winner, a PAQ-8 variant including its code, sat at 1.28 BPC, and mainstream compressors like zip were above 2. A pure neural model had closed to within striking distance of a dedicated compressor. The generated four-page sample shows the network learned a vocabulary and a subword model, inventing plausible-looking words like "Lochroom River" and "submandration"; it balanced parentheses, quotes, and nested XML tags, and generated Cyrillic, Chinese, and Arabic characters, all over long-range structure spanning thousands of timesteps.
For handwriting, the synthesis network with adaptive weight noise reached -1128.2 nats log-loss, about 31 nats better than unregularized (-1096.9). Knowing what to write cut the mean squared error by 44% versus the prediction network (0.23 vs 0.41). Blind taste tests the author ran during presentations suggest that at least some unbiased samples cannot be told apart from real handwriting by eye.
This is the foundational specimen of autoregressive neural generation. Every large model today that emits text, code, or tokens one at a time runs the same recipe: predict the next point, feed it back, predict again. The mixture density network for continuous outputs and the soft window for alignment grew into continuous-output modeling and attention. Biased and primed sampling are the earliest forms of temperature control and conditional generation. Its value to practitioners is not that you would run it now, since the architecture has long been surpassed, but that it states the idea, generation as next-step autoregression with the model as its own training data, more clearly than anything else, with the most intuitive demo. The handwriting section was for a time the most widely circulated evidence that neural networks could generate coherent long-range structure.
The author is candid. Generated text does not make sense beyond short phrases, and he writes that expecting meaningful language from a machine that has never been exposed to the sensory world to which language refers is futile, a line that nearly anticipates the later turn toward multimodal and embodied language learning. The handwriting makes mistakes humans would not: missing, confused, or garbled letters, worse on uncommon words, because the implicit character-level language model is weak on rare vocabulary. There was no independent test set for handwriting; all results were recorded on the validation set, which the author admits may be "somewhat overfit." In the synthesis experiments digits and most punctuation were collapsed into a generic "non-letter" label, which he calls an oversight. The PTB word-versus-character comparison is unfair on parameter count. And every number is a 2013 single-model result, useful only as historical reference today.