Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers
Tianze Xia, Xiaoling Zhou, Ruya Lan, Yiqiao Liao, Chengru Song, Shiming Xiang
cs.CV
2026-10-21
ALLOT sets each layer-step density from final-video error, with no training. On HunyuanVideo, matched-time XAttention gains 11.8 dB PSNR; nine settings tie or beat their defaults.
Full 3D attention in a video Diffusion Transformer (DiT, a diffusion model whose denoiser is a Transformer) scales with the square of the token count. On one GPU, Wan2.1-14B needs about an hour for a 720p, 81-frame clip, and attention is more than half of that time.
Block-sparse attention already skips most query-key blocks. The open choice is how much density to leave in each layer at each denoising step. Published rules keep the first layer dense, or the first 10-30% of steps dense. Automatic allocators instead trust attention mass, Hellinger distance, or the error of a single denoising step. A rule tuned on one model and one block selector does not travel. Those local scores also rank layers differently from final-video error, and following them can lose to a uniform density.
The spread is large. On Wan2.1-14B, the one-step error of layer 0 is 154 times the median layer, yet the density needed to cover 95% of attention mass ranks that layer 22nd of 40. On HunyuanVideo the most sensitive layer is layer 1, at 35 times the median, while attention-map metrics rank it about 47th to 49th of 60.
A cell is one pair: a layer and a denoising step. Its sensitivity is the damage to the finished video when that cell is sparsified to density b, measured against a fully dense run. ALLOT (Allocating Layer-step Latency by Output Tests) never trains and never replaces the block selector or the kernel. It only builds a density table.
A cell probe separates layers inside one step. Only layer ℓ is sparsified; every other layer stays dense. One forward pass at noise level t records the MSE between this velocity prediction and the dense model's. The grid is four OpenVid videos, eight densities, and nine noise levels. Because layer order changes with the selector, probes are taken under the selector used at deployment.
A trajectory gain separates steps. The 50 denoising steps are split into ten windows of five. Each run sparsifies every layer inside one window to density 0.30 and leaves the other steps dense, then records pixel MSE against the dense video. The gain w(s) is that window's final error divided by the sum of probes inside the window. One-step error misses the amplification that later steps apply. On Wan2.1-1.3B the first window's final MSE is 463 times the last window's (PSNR 10.71 versus 37.37), while the one-step errors differ by only 6.4 times. On Wan2.2 the one-step errors even reverse the two windows. Estimated sensitivity is w(s) times the layer's probe at that step's noise level.
The budget is wall-clock, not FLOPs. r(b) is the sparse kernel's time divided by dense FlashAttention on the same shape. Below a crossover density the sparse kernel is faster; above it the solver marks the cell dense. Lagrangian relaxation then picks a density for every cell so estimated error is minimized and measured attention time stays inside the budget. Warm-up length and which layers stay dense fall out of that table.
At inference every prompt reads the same table. Density 1 uses the original dense attention. Anything else keeps the original selector, cuts it at the table density, and runs the original kernel. Resolving a new budget takes about 4 seconds on CPU. Measuring a model takes roughly 6 to 23 GPU-hours, and the two experts in Wan2.2 take more than 30. Rankings barely move across inputs (median Spearman about 0.80 to 1.00), which is why one calibration is reused.
Fidelity is distance to the dense video at the same prompt and seed, not a score of absolute quality. The set is 32 VBench prompts. A win needs the lower end of the paired 95% PSNR interval above zero and a lower mean LPIPS. VBench is not part of the verdict.
With XAttention's selector and kernel fixed, Wan2.1-1.3B at 480p x 81 frames:
| Method | PSNR | LPIPS | Wall-clock |
| Default threshold | 18.14 | 0.268 | 148.7 s, 1.24x |
| HEART (EBC) | 19.43 | 0.221 | 151.7 s |
| FAST | 22.75 | 0.141 | 152.9 s |
| DFSAttn-style | 24.86 | 0.107 | 151.3 s |
| ALLOT | 26.39 | 0.089 | 151.3 s, 1.21x |
Matched to the default threshold's runtime, a second ALLOT table scores 25.37 against 18.14, a paired gain of 7.23 dB on all 32 prompts. In the table above, ALLOT is 1.53 dB above the DFSAttn-style rule and 7.31 dB above Efficient-vDiT at 19.08. Setting the trajectory gain to 1 drops PSNR to 19.34, down 7.05 dB, in line with a DiTFastAttn greedy rule that sees only one-step error (19.38).
On HunyuanVideo at 480p x 129 frames with the same XAttention path, PSNR moves from 15.65 to 27.49, a paired gain of 11.84 dB. ALLOT takes 481.4 seconds and the default 481.3.
Dropped into SpargeAttn, Radial, SVG2, and SVOO, ALLOT wins six of nine published settings and ties the other three. SpargeAttn on Wan2.1-1.3B at 480p goes from 22.67 to 27.70 dB. SVOO on HunyuanVideo at 720p x 129 frames goes from 27.55 to 30.93 dB and from 1.75x to 1.90x end to end. The ties are Radial on Hunyuan (28.08 down to 27.53) and, on Wan2.1-1.3B at 720p, SVG2 (24.25 to 25.60) and SVOO (30.08 to 30.73), where PSNR rises and LPIPS gets worse.
With mean-pooled top-k against a uniform table that keeps the first 15 steps dense, paired PSNR gains on three Wan models are 6.40, 6.38, and 6.50 dB (19.59 to 25.99, 24.87 to 31.25, 19.36 to 25.86). A stronger baseline that also freezes layer 0 and retunes warm-up still trails by 2.55, 0.76, and 1.67 dB. Across 80 density tables, predicted rank and measured final-video error agree at Kendall tau 0.83.
The piece that plugs into an existing video stack is the allocation table. Selectors and kernels from XAttention, SpargeAttn, SVOO, SVG2, and Radial stay. Speedup barely moves on most published pipelines. The gain is a video closer to the dense reference. Quality and speed move together mainly for SVOO on HunyuanVideo.
A hand rule that keeps the first 30% of steps dense and then locks density at 0.30 does not match the solved tables. The cells that stay dense longest are the first few layers plus some deeper ones. Density falls across several middle steps, and the shape changes with the budget. At 30 steps, CFG 7.5, or with an Euler sampler, the same measurements still beat the default. A new selector does require new probes: layer rankings correlate only 0.48 to 0.83 across selectors.
On Wan2.1-14B the margin over the strengthened baseline is 0.76 dB. Once a hand rule already protects the sensitive layers, what remains is a small step. XAttention on the 1.3B model and on HunyuanVideo is outside that range.
Calibration is once per model and selector, and a new selector means new probes. The additive, layer-separable estimate ranks cells better than it predicts magnitudes. When a single layer is sparse for all 50 steps, the gap between the most and least sensitive layers is predicted at 9.3x and measured at 4.8x on Wan2.1-1.3B, and predicted at 146x versus measured 7.7x on the 14B model. One density per cell also replaces per-head, per-input thresholds. On SVOO with Wan2.1-1.3B at 720p, splitting blocks by each head's default density is 1.44 dB worse than giving every head the same fraction, so the equal split is what gets used.
Probes are measured while every other layer is dense. At deployment the other layers are sparse too. A correction factor gains 0.46 dB on HunyuanVideo and loses 6.09 dB on Wan2.1-1.3B, so no table in the paper uses it. Error pattern matters as well as energy. Replacing the real sparse residual with random signs or Gaussian noise of the same magnitude hurts the video far less. On Wan2.1-1.3B, first-window top-k, PSNR to the dense reference is 10.71 for the real sparse error, 29.57 for random signs, and 25.68 for Gaussian noise.
Every fidelity number asks how close the clip is to the dense run. VBench moves both ways, usually by a little. SVOO on Wan2.1-1.3B at 720p goes from 72.12 to 71.45, and the verdict ignores that. The prompt set has 32 items, and the 14B top-k comparison uses 16. Switching a threshold for a density cutoff is not neutral: on SVOO with Wan2.1-1.3B at 480p x 33 frames it costs 3.00 dB. Among the nine integrations, Radial on HunyuanVideo has a lower mean PSNR and is still counted as a tie under the interval rule.