Hierarchical Autoregressive Image Models with Auxiliary Decoders
Jeffrey De Fauw, Sander Dieleman, Karen Simonyan
cs.CV, cs.LG, stat.ML
2019-03-07
Auxiliary decoders learn discrete codes that strip local detail; autoregressive priors on top yield coherent 128×128 and 256×256 ImageNet samples, matching SPN, trailing BigGAN.
Autoregressive image models such as PixelCNN train on likelihood and excel at texture and detail, but their samples fall apart at large scales: locally plausible, globally incoherent. Convolutions bias toward local interactions, and the likelihood loss rewards local correlations far more generously than long-range structure, so that is where the capacity goes. GANs of the period already generated 512×512 ImageNet images but drop modes; likelihood models do not drop modes, yet lagged on resolution and coherence. This DeepMind paper reframes the gap as a representation problem: strip local detail from the input first, let autoregressive models handle the skeleton that remains, and stack encoders into a hierarchy of progressively more abstract codes.
The hard part is learning codes that drop local detail yet support high-fidelity reconstruction. The obvious design, an autoencoder with an autoregressive decoder trained end to end, suffers pathologies the paper dissects, all tracing to teacher forcing (ground-truth context in training, own predictions at inference) colliding with the autoencoder objective: slowly varying information such as global colour goes unencoded while sampling deviations accumulate into colour drift at inference; noise gets encoded instead, since the decoder cannot predict it from preceding pixels; and latent capacity that is too high removes the incentive to use autoregressive connections, while capacity squeezed by a KL term triggers posterior collapse.
The fix is auxiliary decoders: a separate decoder exists only to supply the encoder with a learning signal, is then discarded, and is replaced by a separately trained conditional autoregressive decoder (a local PixelCNN plus a modulator mapping codes to per-layer biases). Two variants:
Encoder outputs pass a VQ bottleneck (snapped to a learnable codebook), bounding code information, stabilising prior training and preventing posterior collapse; encoders downsample so higher levels have lower spatial resolution. All three networks can train simultaneously as long as the autoregressive loss is not backpropagated into the encoder, with negligible effect on reconstruction quality.
128×128 images compress to single-channel 8-bit codes (256 bins) at 32×32, a 48× compression, with PixelSNAIL-style gated PixelCNN priors plus masked self-attention; 256×256 stacks two autoencoders, 192× total.
| Auxiliary decoder | IS | FID | Joint NLL |
| Feed-forward | 18.10 ± 0.96 | 44.95 | 3.343 bits/dim |
| MSP | 17.02 ± 0.79 | 46.05 | 3.409 bits/dim |
The authors decline direct NLL comparisons with earlier papers, citing preprocessing differences. IS and FID trail adversarial models badly; the paper's own framing puts them in the same ballpark as PixelCNN on 32×32 ImageNet from Ostrovski et al. (IS 8.33, FID 33.27). Human ratings (1 to 5, twenty classes, 128×128; real images score only 2.941, since resolution caps realism):
| Model | Average rating |
| BigGAN (high truncation) | 2.113 |
| BigGAN (low truncation) | 1.874 |
| SPN | 1.452 |
| Hierarchical (MSP) | 1.431 |
| Hierarchical (feed-forward) | 1.412 |
Pairwise, MSP samples win 50.57% against SPN, 22.89% against BigGAN (low truncation), and 5.39% against real images. The 256×256 models ship samples but no metrics; lowering sampling temperature from 1.0 to 0.97 or 0.98 yields good samples more consistently. Ablations: deeper auxiliary decoders make codes less predictable, larger masks make them more predictable; the MSE decoder preserves local structure better (text on a box), MSP preserves texture presence better (a bird's body, wallpaper), and both vary in colour and texture across samples, confirming those details never entered the codes.
The paper turns "likelihood models cannot do large images" into a representation-learning problem: codes absorb local detail, priors handle large-scale structure. Concurrent work by Razavi et al. takes a closely related route, multi-scale VQ-VAE plus autoregressive priors, discussed in the paper; their pixel-level decoder is feed-forward, faster to sample but somewhat blurry. The paper itself places MSP next to word2vec, ELMo and BERT; for image tokenizer training today, masking the local patch to force long-range structure remains a usable objective. Temper expectations: 50.57% against SPN is a coin flip, and 22.89% against BigGAN is a clear loss. The contribution is the pathology analysis and the training recipe, not a leaderboard entry.
Stated by the authors: each level's likelihood lives in a different feature space, so levels are not comparable, and the joint NLL is dominated by the pixel-level conditional term, making it uninformative; pixel-domain likelihood is the wrong yardstick for large-scale structure anyway, leaving no unified metric; the human evaluation covers realism only, not diversity; feed-forward versus MSP is left to future work. Gaps the paper leaves open: 256×256 gets no quantitative evaluation; a single autoencoder degrades visibly at that resolution, so the two-level stack's layer counts, mask sizes and compression ratios are hand-configured, with no method given for finding or learning them; and class conditioning hands the prior a strong signal, while the unconditional setting goes untested.