Video diffusion models break physics because RoPE locks motion paths early, arXiv study finds
udmrzn · x · 2026-09-24
A new arXiv paper presents the first interpretability study of motion planning in text-to-video diffusion models, explaining why SOTA systems often violate physics.
Key findings:
- Motion follows a 'first shape, then details' pattern: trajectories are committed during early denoising steps.
- A specific subset of attention heads, identified via cross-attention trajectory patterns and causal head contributions, drives motion planning.
- Rotary Position Embedding (RoPE) causes excessive spatial attention decay, prematurely locking candidate regions into physically implausible positions and suppressing coherent motion in adjacent frames.
Fix: a lightweight architectural modification that scales RoPE frequency across denoising steps. Both training-free and training-based experiments confirm improved physical consistency, without external priors or specialized data.
Related event: Study Explains Why Video Diffusion Models Break Physics(2 posts)→
More from Multimodal
- Generating accurate 3D building models from 150 planning PDFs with new Gemini model — fofrAI · 2026-11-11
- Stanford's DyCraft Uses Agentic Decomposition to Cut Physical Errors in Long-Horizon 4D Scene Generation — StanfordUniversity · 2026-11-11
- Midjourney SREF pick: --2523103199 for loose pencil-sketch portraits — tisch_eins · 2026-11-11
- H3 Image Tips: 16MP via ComfyUI Workflow Plus Detailed Prompts Yields Best Results — gatortux · 2026-11-11
- Gemini-generated endless loop of quantum cryostats self-assembling — hapticdata · 2026-11-11
- Dad brings his daughter's crayon drawings to life with AI for $1.75, and it's unsettling — Illustrious_Ad_4292 · 2026-11-11