Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms
Yueyan Li, Haibo Wang, Caixia Yuan, Xiaojie Wang
cs.CV, cs.LG
2026-09-20
BUPT traces illegal Wan2.1 motion to RoPE locking candidates too early. Scaling RoPE in the first five steps lifts VideoPhy PC from 29.45% to 39.94%, 62.68% with LoRA plus rewrite.
Video diffusion already produces pretty motion. Basketballs still bounce in mid-air, float against gravity, or freeze. Typical fixes bolt on a physics engine, rewrite the prompt, or add physics-heavy data. Those can raise scores. They do not say at which denoising step the model picks the wrong path.
This paper narrows to solid dynamics: how Wan2.1-T2V assigns an object a position in every frame, and why that assignment can violate mechanics.
The workhorse is Wan2.1-T2V-1.3B with 50 denoising steps. The running example is a basketball falling onto a wooden floor against a white background. The 14B model is not better at physics, so the main analysis stays at 1.3B.
Cross-attention is the only door for prompt semantics into the video latent. Maps onto the object token ("basketball") go from noise at step 1, to several highlighted candidate regions per frame at step 3, to a settled position around step 5, to a clear trajectory by step 7. Attention entropy and support quality (overlap with the final object mask) both jump near step 5.
A trajectory-shaped map is not enough to call a head a motion planner. Causal patching scores each head's contribution; heads are then split by convergence speed. Zeroing the small set that both shows a trajectory pattern and scores high collapses the path. Early layer-0/1 heads look more like appearance init: remove them and the object does not even move. Heads with a clean trajectory pattern but near-zero contribution barely change the path when ablated.
Self-attention is the coordination layer. Most heads do not follow the trajectory. They pin the same spatial patch across frames. The paper calls this RoPE's spatial anchoring: rotary encodings on height and width decay the dot product as spatial offset grows, so frames prefer to align "the same pixels." Early candidates compete. A few frames lock first. Neighbors are pulled toward that lock. If the early winner is an illegal mid-air point, a physically better but spatially farther candidate is suppressed, and the clip shows a hover or an aerial bounce.
The fix is small. In the first few denoising steps, multiply height and width RoPE frequencies by λ<1 (training anneals toward 0.75 by default), which slows spatial decay and keeps other candidates in play. Training also upsamples the first 10% of timesteps. LoRA is applied mainly to attention.
VideoPhy has 343 test cases. Metrics are semantic adherence (SA) and physical commonsense (PC).
| Method | Overall SA | Overall PC | Solid-solid PC |
| Wan2.1-T2V-1.3B | 53.64 | 29.45 | 24.52 |
| Prompt rewrite | 80.47 | 44.31 | 28.88 |
| LoRA | 55.69 | 35.57 | 27.95 |
| VideoREPA | 56.56 | 37.90 | 30.61 |
| Modified RoPE | 57.43 | 39.94 | 32.60 |
| LoRA + modified RoPE | 64.72 | 41.98 | 35.39 |
| LoRA + RoPE + rewrite | 87.46 | 62.68 | 50.61 |
Prompt rewrite mostly lifts SA. RoPE scaling beats standalone LoRA and VideoREPA on PC, with a clearer edge on solid subsets; fluid-fluid motion is smaller, and so is the gain. On the basketball case, a training-free λ still needs hand tuning. After LoRA, a fixed λ helps other random seeds. Tuning attention and leaving the FFN alone improves physics without hurting aesthetics, which fits the claim that the failure lives in attention, not in feed-forward memory.
Most physics-aware video work adds data or an external simulator. This paper locates a failure in spatial RoPE during the first few denoising steps, and moves PC with a single scale factor. Part of "anti-physics" is an inductive bias of the position encoding, not only missing physics footage. The scale stacks with prompt rewrite: rewrite helps instruction following, the scale delays premature lock-in.
The recipe is most direct for Wan-style DiTs. Whether other RoPE video models respond the same way is left as future work.
The mechanistic story is built around one falling-basketball prompt and solid dynamics. Optics, heat, and materials are almost untouched. λ still has to be chosen, and the training-free setting is seed-sensitive. VideoPhy SA/PC are binary automatic scores, not millimetre trajectory error. The 14B model appears only as appendix clips, with no full table aligned to 1.3B. Image-to-video and few-step autoregressive diffusion are named and not run. Causal scores patch velocity prediction; that is a proxy for "physical correctness," not for final pixels.