Improving Transformer Models by Reordering their Sublayers
Ofir Press, Noah A. Smith, Omer Levy
ACL 2020
cs.CL, cs.LG
2019-11-10
Rearranging Transformer sublayers so attention sits low and feedforward sits high cuts WikiText-103 perplexity from 18.70 to 17.96, at no extra parameter or training cost.
A Transformer layer pairs a self-attention sublayer (s) with a feedforward sublayer (f), and stacking layers gives the interleaved pattern s f s f s f… that every model since "Attention is All You Need" (2017) has used, BERT and GPT-2 included. Ofir Press, Noah Smith (UW and Allen AI), and Omer Levy (Facebook AI Research) ask a blunt question: what makes us assume this ordering is optimal? Nothing in theory says it is. The paper does not add a new module. It reshuffles the existing ones and watches what happens.
They start with random search. Holding the parameter budget fixed (16 s plus 16 f), they permute all 32 sublayers, train from scratch, and read perplexity on WikiText-103. Seven of twenty random models match or beat the baseline average, and the best hits 18.19 against the five-seed baseline average of 18.65. A better ordering almost certainly exists.
Slicing each model into top and bottom halves by parameter count reveals a pattern: models that beat the baseline pack more self-attention into the bottom half and more feedforward into the top half. This motivates the sandwich transformer: k pure self-attention layers at the bottom, k pure feedforward layers at the top, interleaved in between. Written as s^k(sf)^(n-k)f^k, with k the "sandwich coefficient." k=0 recovers the original Transformer; larger k pushes both ends toward the extreme.
Why does it help? The paper is candid: they do not know. An attention-distance metric does confirm that sandwich and interleaved models learn genuinely different attention distributions, so the reordering changes what the model computes.
The headline is on the WikiText-103 test set:
| Model | Perplexity |
| Baseline (Baevski and Auli, single run) | 18.70 |
| Transformer-XL | 18.30 |
| Sandwich16 (coefficient 6) | 17.96 |
The sandwich beats the baseline by 0.74, roughly double the gap between the baseline and Transformer-XL (0.40), at zero cost in parameters, memory, training time, or epochs, with hyperparameters untouched. Across five seeds on the dev set the sandwich scores 17.98 ± 0.10 against the baseline's 18.65 ± 0.34, with smaller variance.
Transfer to other tasks shows diminishing but positive returns. On Toronto Books (word-level) the sandwich reaches 10.83 against a baseline of 11.89 ± 0.35, a 1.06 drop. On character-level enwik8, layered onto the Adaptive Span model, it hits 0.968 BPC, matching the then-state-of-the-art Compressive Transformer (0.97) without Transformer-XL's slow recurrent attention and with fewer parameters (209M vs 277M). Machine translation (WMT14 En-De) is the holdout: BLEU stays in the 28.4 to 28.8 range whether the encoder or decoder is sandwiched, against a baseline of 28.74 ± 0.15.
The practical pitch is simple: a free, drop-in improvement. Reorder the sublayers and training and inference cost stay flat while perplexity drops. The authors also frame it as lightweight, human-in-the-loop architecture search that trains a few dozen models rather than the thousands classic NAS demands.
It also punctures an assumption that sat unchallenged for years: interleaved sublayers are not optimal, which leaves room for task-specific orderings.
The biggest gap is mechanistic. The authors admit they cannot explain why reordering helps, only that the learned attention changes.
The gains do not generalize. Language modeling improves; machine translation does not budge. The further from the original setting, the smaller the gain, and the coefficient k must be tuned per task on the validation set rather than fixed once.
In hindsight the pattern was never adopted by mainstream large models. Llama and the GPT family still interleave. A 0.7 perplexity gain that holds only for language modeling was not enough to move infrastructure. Read it as a tidy architecture-search finding and a useful engineering trick, not as a blueprint for the next generation.