Linear readout replaces the VAE decoder and cuts Wan warping error 23%

Decoding Without a Decoder: Direct Pixel Prediction from Diffusion Transformers

Hansheng Chen, Shengqu Cai, Jan Ackermann, Rayhan Zirvi, Gordon Wetzstein, Xinchen Yan, Leonidas Guibas

cs.CV

2026-11-17

DWD adds a linear head that predicts pixels only at the last step, removing the VAE decoder. Wan2.1 14B warping error drops 23%; pixel-flow ImageNet FID falls from 1.55 to 1.43.

What problem this solves

Latent diffusion denoises inside a compressed latent with a DiT, then a convolutional VAE decoder turns that latent into pixels. The decoder is another network to deploy, and it costs time. On an H100, 4-step Wan 1.3B spends 2.8 seconds sampling a 5-second video and 1.9 seconds decoding it, roughly 70% overhead. Faster decoders have already come close on quality, so much of that compute looks redundant. Getting pixels out of a latent DiT does not require a full decoder network.

Pixel diffusion has no decoder. The denoising head is still an ℓ2 fit to a conditional mean, and that average wipes fine texture. Perceptual losses on the same head pull detail and diversity in opposite directions. A shared-head ablation moves AnisoFlow ImageNet FID from 1.55 to 1.71, worse than the baseline. The two outputs have to be separate.

Method

DWD adds a linear head beside the denoising head. Both read the last-layer tokens. The denoising head keeps driving the sampler. The linear head runs only on the final forward pass, maps each token to a pixel patch, and the patches are assembled into an image or a video. The usual spatial factor is 16, so a token covers 16×16×3 pixels. Video adds a temporal factor of 4, and one token covers four frames.

On a frozen model, a least-squares probe from those tokens already reconstructs pixels better than a probe from the denoised latent. Figure 3 shows relative RMSE reductions of about 12% to 51%. Finetuning the last K blocks closes the rest. Large text models typically use K of 4 to 8, and training also updates both heads and the final modulation while the prefix stays frozen. Gradients stop at the tail, so training is up to 3× faster than a full backward pass.

The loss is a pixel term plus trajectory anchoring, with separate timestep ranges. The pixel term uses L1, LPIPS, and an adversarial loss. The discriminator is StyleGAN-T, scaled by gradient norm, and turned on late. Its timesteps are low noise, uniform on 0 to 0.1 before the shift, matching the final step. Anchoring matches the trainable denoising head to a frozen copy of the original velocity, on 0.04 to 1 before the shift. The two losses share one pass through the frozen prefix and split only in the last K blocks. The final step skips classifier-free guidance. That step does not move the trajectory, and guidance is only there to steer it.

The final time has to be above 0. In Figure 4, t=0 means a fully denoised input and the FID gain is gone. A little noise left in the input is what lets the head restore texture the latent had smoothed away.

Results

On ImageNet 256, removing the decoder nudges latent FID down. REPA goes from 1.38 to 1.36 and IS from 306.2 to 312.5, cutting 622 decoder GFLOPs, about 2.6× one denoiser evaluation. REPA-E goes from 1.12 to 1.06. JiT-H/16 goes from 1.90 to 1.85. AnisoFlow-H/16 goes from 1.55 to 1.43, IS from 300.1 to 303.5. The paper calls 1.43 a new best among pixel flow models. REPA-E with DWD still sits at 1.06 in the same table. The shared head pushes AnisoFlow to 1.71. A small convolutional residual does not beat the linear head consistently.

Final-step FLOPs in the tables are not a pure decoder comparison. Baselines include a guidance pair, and DWD's last step is conditional only.

SetupMetricResult
REPA to DWDImageNet FID1.38 to 1.36
REPA-E to DWDImageNet FID1.12 to 1.06
AnisoFlow-H/16 to DWDImageNet FID1.55 to 1.43
Shared headImageNet FID1.55 to 1.71
AnisoFlow T2ICOCO pFID36.59 to 32.26
Wan2.1 14BWarping error ×10^-31.52 to 1.18
Wan 14B, 8-bitWarping error ×10^-31.69 to 1.20
Wan 1.3B, 4-stepEnd-to-end latency4.661 s to 2.757 s

At 1024 text-to-image, FLUX.2's latent is wide: 128 dimensions per token for a 768-dimensional pixel patch, against 64 for a 3072-dimensional patch in Wan. Decoding it was already easy. The base model ties the baseline on COCO FID, pFID, HPS, OneIG, and LongText, and drops 10.47 TFLOPs of decoder work per image. The 4-step variant moves COCO FID from 21.50 to 21.92 and pFID from 25.52 to 25.33, with the other scores tied, even though its final step is at t=0.7 after the shift. Pixel-space AnisoFlow-L/16-T2I, after 3,000 steps, moves COCO FID from 28.86 to 27.54 and pFID from 36.59 to 32.26. Skin and hair texture come back, and preference scores stay put. Without the adversarial term, pFID slips more and the other metrics stay close.

Wan2.1 14B must predict four frames from each token. On 81-frame 832×480 video, warping error falls from 1.52×10^-3 to 1.18×10^-3, down 23%. The score aligns neighbors with optical flow, masks occlusions, and measures the leftover pixel error, which is the frame-to-frame warp of mesh and pipes. VBench temporal flickering is near the ceiling for both and misses that. Other VBench axes and HPS stay tied, while dynamic degree falls from 0.736 to 0.711. A decoder finetuned on the same data with the same pixel losses only reaches 1.39. With FP8 weights and 8-bit SageAttention2, the baseline rises to 1.69 and DWD stays at 1.20, a gap of about 29%.

On 4-step Wan 1.3B, latency falls from 4.661 seconds to 2.757 seconds, peak memory from 19.22 GB to 16.15 GB, and warping error from 2.84×10^-3 to 2.07×10^-3. 4-step FLUX.2 only moves from 0.776 seconds to 0.695 seconds. AsymFlow converts the latent trajectory into pixel diffusion and drops LongText to 0.777. DWD holds 0.908, at about 350 H100 hours against about 1,100.

Why it matters

At four sampling steps, decoding is a large share of wall time. DWD replaces it with a linear readout on the network tail, and the anchoring loss keeps the original trajectory, so same-seed content stays. The 1.69× end-to-end speedup on 4-step Wan 1.3B is the clearest deployment number. Under 8-bit inference the warping error is also more stable than with the original decoder.

The FID of 1.43 is the cleaner result. No weak decoder is available as an excuse. A separate head can take perceptual losses without rewriting the sampler. FID moves of 0.02 and 0.06 on latent ImageNet are small. The visible differences are video flicker and texture on pixel models.

Limitations

The paper lists three. Gains shrink once the decoder is already strong, and FLUX.2 ties its baseline. A linear head is not shown to be enough for spatial patches above 16×16. MiniMax-H3 uses 32×32 and up to four frames, so one token would emit 12,288 channels, more than twice a hidden size of 5,376. The appendix treats that setting as preliminary and says the linear head runs out of capacity. Pixel losses were not tuned per model.

Dynamic degree on Wan 14B falls from 0.736 to 0.711. The paper reports no visible loss of motion and does not measure motion magnitude on its own. Most VBench axes tie, so the quality claim rests on warping error and side-by-side frames. The final timestep is hand-picked, and t=0 removes the gain. There is no comparison against fast decoders such as FlashDecoder, so the experiments do not decide whether a linear head beats a smaller decoder. The Wan 14B finetune is estimated at 4,300 H100 hours.

Terms

Source

What people are saying

Related papers

All paper explainers