MPEG-Diffusion: Mixed-Resolution Video Generation
Dennis Menn, Tianpei Gu, Guande He, Aryan V S, Yanke Song, Nan Liu, Desai Xie, Xun Huang, Chenfeng Xu, Diana Marculescu
cs.CV
2026-11-09
On Wan2.1 14B, MPEG-Diffusion uses 30% of the tokens, denoises 4.8× faster, stays near full resolution, and wins over 90% of pairs against prior mixed-resolution methods.
A video DiT spends the same compute on a face and on empty sky. DiT here means a diffusion model whose denoiser is a Transformer. Attention grows with the square of the token count, so resolution and duration get expensive fast.
Mixed resolution copies an old codec trick: keep the important region at high resolution (HR) and the rest at low resolution (LR), letting one LR token stand in for several HR tokens. On video, previous versions of this idea blur detail and leave a seam at the boundary. They also need the HR region before generation starts. Early in denoising the frames are too blurry for a detector to supply that region.
MPEG-Diffusion comes from UT Austin and Roblox and is built on Wan2.1. It runs in four stages. The first k denoising steps treat the whole clip at LR. At step k a linear head reads DiT features and picks a region M. Later steps denoise M at HR and everything else at LR, each at its own resolution. A decoder then merges the two latents into one HR frame. A latent is the compressed feature map the VAE turns back into pixels. By default one LR token covers a 2×2 block, so n = 4. During training the HR region is the 25% of tokens per frame with the highest perceptual label.
The first failure mode is a shared key/value cache. Keys and values (K/V) are what attention looks up. Softmax normalizes over the number of tokens, and one LR token cannot reproduce the effect of the n HR tokens it replaces. Proposition 1 shows that a logit bias on the LR token still makes the equality hold only when those n tokens are identical. Fine-tuning a shared K/V does not close the gap.
Dual-resolution attention keeps a full HR K/V and a full LR K/V in every layer. Each query reads only its own resolution. Linear maps fill the missing entries: one LR token expands to n HR tokens, and n HR tokens collapse back to one LR token. The maps sit before RoPE, rotary position embeddings, so they change content and leave each resolution on its own grid. On the untouched Wan2.1 1.3B the fit is already strong. Average cosine similarity from HR to LR is 0.96 for keys and 0.85 for values; the reverse direction is 0.94 and 0.75. Large models replace the full matrix with a low-rank correction on top of averaging or copying.
The second failure is the decoder. A VAE decoder expects a uniform grid, and each patch depends on its neighbors. Encoding a real video at both resolutions, decoding the two latents separately, and pasting them still leaves a seam. Filling the hole with noise spreads the artifact. The mixed-resolution decoder writes a downscaled copy of the HR latent into the LR hole, upsamples with a learned upscaler, and lets a small 3D convolution predict a residual only on the LR side. The HR region is kept as generated.
The HR region does not have to be supplied. At step k, features from a middle DiT block already mark the subject and follow it across frames. A linear head scores each LR token. Scores are thresholded with that video's own median and 99th percentile. If the scores are flat, the clip stays at LR. One training target is perceptual: per-patch LPIPS between the LR prediction at step k and the ground-truth HR frame. LPIPS is a feature-space distance, and high-frequency background texture distracts it. The other target uses Qwen2.5 to name the subject and Grounded SAM to track it. Inference runs only the linear head trained on that object target.
The DiT, the K/V maps, and a pair of resolution embeddings are fine-tuned together. Each embedding is a vector added to tokens of one resolution, starting at zero. The loss covers only the region a latent is responsible for. The DiT is tuned for 1,250 steps on 64 B200 GPUs. The upscaler and mixer train separately for 4,000 steps. The region head uses about 2,500 clips, with the DiT frozen.
Video tests use Wan2.1 1.3B at 480p and 14B at 720p (704×1280), 81 frames and 50 steps. Baselines are the token-merging methods ToMe and IDM, plus Phase-Aligned RoPE and Foveated Diffusion. Text-to-video has no gaze point, so the last two follow their original setup: a circle moving on a preset path, matched in token count. MANIQA, Imaging, and DOVER use 1,003 MovieGen prompts. VBench Quality uses 944 VBench prompts. Speed is denoising time on one B200, relative to full resolution.
At k = 40, about 30% of full-resolution tokens, MPEG-Diffusion is best or tied on the quality metrics for both model sizes.
| Setup | Method | Speedup | MANIQA | DOVER |
| 1.3B, 30% | Full resolution | 1.0× | 0.371 | 0.778 |
| 1.3B, 30% | Foveated Diffusion | 3.2× | 0.261 | 0.658 |
| 1.3B, 30% | MPEG-Diffusion | 3.8× | 0.345 | 0.762 |
| 14B, 30% | Full resolution | 1.0× | 0.445 | 0.819 |
| 14B, 30% | Phase-Aligned RoPE | 5.2× | 0.293 | 0.757 |
| 14B, 30% | MPEG-Diffusion | 4.8× | 0.397 | 0.794 |
ToMe and IDM collapse at this ratio. DOVER falls to 0.119 and 0.110 on 1.3B, and to 0.069 for both on 14B. Foveated Diffusion at 14B and 30% tokens is also 5.2×, with MANIQA 0.300 and DOVER 0.743. Imaging stays near full resolution: 0.631 versus 0.644 on 1.3B, and 0.644 versus 0.671 on 14B. The DOVER gaps are 0.016 and 0.025. VBench Quality barely sees blur. On 14B, full resolution scores 85.3, pure LR scores 84.4, and MPEG ties Foveated at 83.9.
At k = 25 the 1.3B run uses about 38% of the tokens, and all three mixed-resolution methods sit at 2.8×. MPEG scores MANIQA 0.355 and DOVER 0.769, against 0.285 and 0.701 for Foveated Diffusion. On 14B, about 37% of the tokens, MPEG is slower, 3.3× against 4.2×, because every HR token attends to a full HR K/V. Quality still leads: MANIQA 0.402 versus 0.316 and 0.323, DOVER 0.803 versus 0.769 and 0.761. Cutting the budget to 30% flips the speed comparison. 4.8× beats the 4.2× those baselines reach at 37%.
At 960p with a 240p LR track, one LR token covers 16 HR tokens. The same k = 25 setting speeds up 4.8×, against 3.3× at 720p. DOVER is 0.789 versus 0.806 for full resolution and 0.766 for Foveated Diffusion at 4.7×. MANIQA is 0.367 versus 0.472, a wider gap than at 720p.
The 14B user study has 15 raters and 400 uncurated pairs. Draws are split evenly. At a 37% token ratio, raters prefer MPEG over Phase-Aligned RoPE, Foveated Diffusion, and pure LR in 98.8%, 90.1%, and 69.1% of judgments. Against full resolution, a draw is the most common raw answer at 44%. After the split, full resolution leads 54.8% to 45.2%, while MPEG is 3.3× faster.
On FLUX.2 [klein] 4B at 1024, with 1,000 prompts and 33% of the tokens, Q-Align is 4.60, against 4.49, 4.42, and 4.34 for Foveated Diffusion, pure LR, and Phase-Aligned RoPE. Patch FID versus full-resolution images is 12.97, against 16.40, 15.61, and 14.59 in that same order. GPT-5.6 prefers MPEG in 61.4%, 78.0%, and 83.8% of those pairs. Speedup per denoising step is 1.40×, against 1.14× and 1.11×. An image has only 4,096 tokens, and the baselines run mixed resolution on every step, splitting each attention layer into two calls.
On the 1.3B profile, K/V maps take 209 ms, the mixer 51 ms, and the region head 3.4 ms, together 3.0% of runtime. The DiT itself takes 8.54 s. A shared K/V bends a clock hand that crosses the resolution edge. Dual-resolution attention cuts warping error at that edge from 2.61×10^-3 to 1.96×10^-3, and temporal LPIPS from 4.59×10^-2 to 4.14×10^-2. The spike at the edge drops from about +70% to about +25%. On 100 real clips, the mixer cuts LPIPS near the edge from 0.19 with paste-together decoding to 0.12.
The savings come from spatial redundancy. Token merging, sparse attention, feature caching, and step distillation still emit every region at one resolution. Mixed resolution can sit beside them.
A video model can keep the subject sharp and spend less on the background, and inference does not need a box. Using it means fine-tuning the DiT. On 1.3B the maps, mixer, and region head add about 3% to runtime, so the speedup is mostly fewer tokens. The same k = 25 schedule goes from 3.3× at 720p to 4.8× at 960p.
Images gain less. A 33% token budget buys 1.40×. The mechanism transfers to a text-to-image DiT, and the payoff is smaller than on video.
The region head is trained apart from the DiT. Object labels depend on Qwen2.5 and Grounded SAM, and the paper leaves end-to-end fine-tuning as follow-up work. Clips with no clear subject fall back to pure LR. There is no separate number for landscapes, crowds, or frames full of text.
At k = 40 on 14B, MPEG runs at 4.8× while Phase-Aligned RoPE and Foveated Diffusion run at 5.2×. If the goal is wall-clock time and a seam is acceptable, the older methods are faster at that operating point. In the user study their HR region is a preset moving circle, so part of the preference comes from region choice. Attention and the decoder have separate ablations. The older methods are not compared again under the same self-selected mask.
Raters still lean slightly toward full resolution, 54.8% to 45.2%, with draws at 44%. The linear map from LR values back to HR reaches a cosine similarity of only 0.75, so the fill-in is approximate. The mask is fixed at step k and never updated. The paper does not report how often that fails when the subject appears late. On 14B VBench Quality, pure LR scores 84.4 and MPEG scores 83.9. That metric cannot carry a quality claim.