Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation
Haocheng Xi, Yiming Xie, Hexu Zhao, Yiwen Zhang, Michael Liu, Thomas Creavin, Kurt Keutzer, Xiuyu Li, Zhaoyang Lv, Chenfeng Xu, Haiwen Feng
cs.LG
2026-09-18
VDN adds local Softmax plus frame-wise linear memory to MiniMax H3; 8-step DiT denoising of 14.3s 768p takes 6.70s on eight B200s.
Video diffusion transformers keep attending over long spatiotemporal token sequences at every denoise step. On the profiled MiniMax H3 workload, Softmax attention is more than 85% of denoiser time. Dense Softmax gives every query the full key set, and the cost grows quadratically with length. Language models already compress distant context into a fixed recurrent state with delta-rule linear attention. Dropping that recipe onto video hurts quality: a fixed-size state cannot keep every fine correspondence, token-by-token writes fight a frame whose spatial tokens arrive together, and a randomly initialized linear branch disturbs a Softmax-pretrained residual stream before it becomes useful.
Video DeltaNet (VDN) splits video-to-video attention by temporal role. Nearby frames keep exact Softmax. H3's VAE decodes five latent frames as one chunk, so each query chunk attends to itself plus the previous and next chunks, a 15-frame window that does not cut through the tokenizer's temporal unit. Two boundary anchors add four-way dense links between the first and last latent frames and the rest of the clip, which is a natural global reference for full-clip diffusion, image-to-video, and first-last-frame conditioning.
Context outside the window is a bidirectional linear memory. The forward state summarizes frames before the window, the reverse state summarizes frames after it. The two spans are disjoint, so adding the readouts does not double-count a frame. Before scanning video, the linear branch folds all text tokens into one state and splits it across the two directions so text is counted once. Interactions that involve text or audio stay on Softmax.
The linear write is not token-wise. Video Delta Attention (VDA) updates memory once per frame from all of that frame's spatial tokens. The new state stays close to the decayed memory and fits every key-value pair in the frame at once. The closed form inverts (I+A) in key-channel space, so overlapping keys constrain each other and there is no SANA-WM-style 1/sqrt(U) key scale. A short proposition shows the inherited-state map is non-expansive when write gates are nonnegative and decay gates sit in [0, 1]: a frame update cannot amplify older memory.
Both branches reuse the pretrained QKV projections, with separate gates and output maps. Softmax mass is more peaked inside a short window, so that branch gets a content-dependent sigmoid gate; the linear readout is RMS-normalized, then gated. Adaptation keeps the backbone frozen through three stages: 200 steps of per-layer linear alignment, 500 steps of end-to-end hybrid alignment, then 2000 steps of QKVO LoRA with the linear path and Softmax gates. An eight-step student is then distilled with a DMD2-style objective without the GAN term, initialized from the community MiniMax-H3-Turbo-LoRA and targeted at VDN-H3's own 50-step sampler.
Training uses 10,015 clips at 1344×768, 345 frames, 24 fps (14.375 s). Evaluation is 103 prompts from a fixed third-party set, with no per-model prompt picking.
On quality, eight-step VDN-H3 matches or slightly exceeds 50-step Dense H3 on five no-reference metrics, with gaps from +0.06 to +1.00; four-step FastH3 sits 2.70 to 12.74 points lower. RAFT mean flow is 11.71 pixels versus 11.55 for Dense H3 and 9.19 for FastH3. FIRM-Video instruction following is 2.25 for both, perceptual quality 4.45 versus 4.40, world coherence 1.77 versus 1.84. First-last-frame PSNR is 28.67 versus 28.85, SSIM 0.826 versus 0.833, LPIPS 0.1156 versus 0.1044.
Speed should be split into three parts. At 50 steps on one GPU, B200 backbone time falls from 15.99 s/step (799.6 s) to 6.16 s/step (307.9 s), about 2.6×; H200 falls from 35.35 s/step to 11.16 s/step, about 3.2×. As latent frames grow from 42 to 102, Softmax density falls from 42.1% to 20.0%, and the backbone speedup grows with length. Eight-step distillation cuts one-B200 denoising to 49.3 s; eight-GPU head sharding then finishes a 14.3 s 768p DiT denoise in 6.70 s. The paper reports 14.5× versus 50-step dense H3 on the same GPU count, and 119.3× versus single-GPU Dense H3. Eight H200s at eight steps take 12.5 s.
If the goal is livestream latency on an existing video DiT, this paper changes attention geometry instead of training a linear backbone from scratch. Local Softmax keeps texture and short-term motion; a frame-wise delta memory carries range; distillation and the serving stack do the rest. The 6.70 s headline bundles architecture, eight-step distillation, and eight-GPU sharding. Hybrid attention alone is 2.6× at 50 steps on one B200.
For teams already running H3-like window attention, the systems pieces of VDA are the transferable part: scans composed at VAE-chunk granularity, a register Gauss-Jordan inverse for the small matrix, and overlap with window Softmax. Text and audio remain Softmax, so this is not a fully linearized model.
There is no standalone limitations section. The hybrid applies only to video-to-video attention; text and audio interactions stay quadratic. Quality is measured on 103 fixed prompts, a narrow slice, and there is no matched comparison with Sparse VideoGen or SLA. The 14.5× figure mixes architecture, distillation, and multi-GPU serving; it is not the linear branch by itself. The recipe is tied to MiniMax H3 and a community Turbo LoRA. A tokenizer without five-frame VAE chunks would need a different window and scan. Inverting (I+A) grows with key dimension; the paper only shows it on H3's head width.