Generating Sentences by Editing Prototypes
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, Percy Liang
cs.CL, cs.AI, cs.LG, cs.NE, stat.ML
2017-09-26
Stanford's prototype-then-edit model reaches 26.87 Yelp perplexity versus 39.03 for an LSTM LM, and its edit vectors support sentence-level analogies better than a sentence VAE.
Neural language models in 2017 generated left to right from scratch and collapsed toward generic lines like "I don't know." Raising temperature bought diversity and usually wrecked grammar. People rarely write a complex sentence in one pass. They draft, then revise.
Kelvin Guu, Tatsunori Hashimoto, Yonatan Oren, and Percy Liang at Stanford write that process as an unconditional generative model: sample a prototype uniformly from the training corpus, then let a neural editor rewrite it under a random edit vector. Corpus sentences are already grammatical, diverse in length, and unbiased toward short vagueness. Attending to a prototype turns generation into a local edit, which is a much smaller problem than sampling from a blank page.
On Yelp, 70% of test sentences sit within word-token Jaccard distance 0.5 of some training sentence, even though almost none repeat verbatim. Most of the test distribution is a small transformation of something already in the data.
The likelihood of a sentence x marginalizes over every prototype x' and edit vector z, which is intractable. Training uses two lower bounds.
The sum over prototypes is restricted to a lexical neighborhood: training sentences with Jaccard distance below 0.5, precomputed with LSH and minhash. In 250 human-rated pairs from those neighborhoods, 35.2% were judged exact paraphrases, 84% at least roughly equivalent, and only 7.2% negated or changed topic. Lexical neighbors in a review corpus are usually semantic neighbors as well.
The expectation over z uses an ELBO with an inverse neural editor q(z|x',x) that proposes edit vectors likely to map x' to x. The editor is a 3-layer bidirectional LSTM encoder plus a 3-layer LSTM decoder with attention; z is concatenated into the decoder at every step. q sums GloVe vectors of inserted words and of deleted words, concatenates those two sums, then perturbs direction with von Mises-Fisher noise and magnitude with uniform noise. The prior draws length from Unif(0,10) and direction uniformly on the sphere. The concentration κ sets the KL term; at κ=0 the KL can be driven near zero.
At test time, sentences with no neighbor are linearly mixed with a standard NLM using weight 0.1, equivalent to occasionally sampling a dummy prototype that can be edited into anything.
| Model | Yelp perplexity | BillionWord perplexity |
| KN5 | 56.55 | 78.36 |
| NLM (LSTM) | 39.03 | 55.15 |
| NLM+KN5 | 37.31 | 47.47 |
| NeuralEditor (κ=0) | 26.87 | 48.76 |
On Yelp the editor sits about 12 points below the LSTM LM and below the best NLM+KN5 ensemble. BillionWord has fewer near neighbors; the editor still beats a standalone NLM and does not beat the ensemble. Ensembling retrieval-without-edits (Memorization) barely helps.
On 400 human ratings, NeuralEditor at temperature 1 matches a temperature-tuned NLM on grammaticality and corpus plausibility, with higher unigram entropy. Lowering temperature slightly improves grammar without much diversity loss, because diversity is injected by the sampled prototype. A greedy decoder still cannot collapse to one generic sentence.
For syntactic transforms from He et al., identity has the highest probability (0.33 nats/token), to-clause reordering 1.62, active/passive 3.27, random shuffle 4.42, and editing to a random sentence 6.07.
Random-walk edits stay related: only 1% of steps were judged unrelated to the prototype. A sentence VAE often copies the input exactly or jumps to an unrelated line. Transferring the same edit vector onto a new sentence for word-level analogies yields top-10 accuracy close to GloVe on the simpler lexical analogy task; the SVAE's top-k accuracy is near zero.
This is an early, fully specified retrieval-augmented language model for the unconditional case: train by marginalizing over output-side neighbors, generate by drafting then editing. Later RAG, retrieval-augmented LMs, and edit-based paraphrase or style transfer share that skeleton.
For controllable generation, an edit vector is easier to use than a whole-sentence latent. The paper's own comparison says local variation over edits is easier to model than global variation over sentences.
The LSTM backbone and Jaccard neighborhoods are dated. The modelling idea is not: a generator does not have to sample from empty context if the corpus already contains good sentences to start from.
There is no dedicated limitations section. Reported perplexity is a neighborhood lower bound plus smoothing, so cross-model gaps should be read with that caveat. The 0.5 Jaccard cutoff is arbitrary, and semantic evaluations are concentrated on Yelp; open-domain text would look worse. On BillionWord the method does not beat the n-gram ensemble, which shows how much it depends on test sentences sitting near the training set. The SVAE baseline is the 2016 sentence VAE, without later tricks for posterior collapse. Human studies describe relative patterns and do not report a single mean score that can be quoted as a headline number.