Self Gradient Forcing adds missing memory-writing gradients for long video extrapolation
Junhao Zhuang · hf · 2026-07-23
Self Gradient Forcing adds missing gradient supervision for long video extrapolation
The paper identifies a historical context-gradient gap in autoregressive video diffusion trained with Self Forcing: future frames consume past key-value cache as frozen rollout state, so later losses cannot tell the model how earlier latents should be written into more useful memory.
What SGF does
- Pass 1: run a no-gradient autoregressive rollout that matches inference, and record the generated context plus the noisy latents at a sampled denoising exit step.
- Pass 2: reconstruct context gradients in parallel for that exit step, recomputing KV representations and future-to-context causal attention.
- This provides memory-writing supervision without full backpropagation through the serial rollout.
Reported results
- Stronger native long-video extrapolation than Self Forcing across long-horizon frame-wise and chunk-wise settings.
- Improvements are especially visible in subject identity, background/layout consistency, and temporal stability.
- With only a 5-second training window, the method can extrapolate to videos lasting several minutes.
The authors say code and models will be released.
More from Research
- Kimi K3 finds 32 of 86 vulnerabilities for just $19.37 in benchmark run — zeeg · 2026-07-23
- A chart comparing AI detectors says most are barely better than a coin toss — burkov · 2026-07-23
- CoLT teaches multimodal models to think in latent steps and cuts inference 10.1x — jiqizhixin · 2026-07-23
- AI agent finds a refutation to a 30-year-old graph theory conjecture in Slack — veggie_eric · 2026-07-23
- Duke study says better memory, not a bigger model, lifted ARC-AGI-3 scores by 18 points — imjustnewatai · 2026-07-23
- Trace builds a reproducible RLVR environment for 11 visual reasoning domains — Md Tanvirul Alam · 2026-07-23