Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning
Haodong Li, Shaoteng Liu, Tianyu Wang, Chongjian Ge, Sihui Ji, Jiahan Zhang, Xin Lin, Haolin Lu, Zhe Lin, Manmohan Chandraker
cs.CV
2026-08-11
LDR turns latent transitions into kinematic integration and learns only the high-order residual, cutting the ID-OOD error gap over 20× vs a DiT baseline, at 26× fewer params and 143× faster.
Video diffusion models can render the next frame convincingly without obeying the laws of motion. The authors' thesis is that these models learn "what the world looks like" rather than "how it evolves." Once a test scenario leaves the training distribution, they collapse to reproducing the closest training sample instead of rolling forward under the true dynamics.
The authors want to separate two things: a video generator (looks right) and a video world model (knows the laws and extrapolates). Judging whether a model understands dynamics is pointless on in-distribution data, where everyone looks right; the test is whether it extrapolates the learned law to unseen initial conditions. They build five tasks on a white-box physics simulator (PhyWorld): uniform motion, parabola, collision, bouncing, and looming, each with in-distribution (ID) and out-of-distribution (OOD) initial conditions that follow the same motion laws. OOD simply pushes speed, radius, and similar parameters outside the training range, to expose whether the model learned the law or memorized pixels.
The method, Latent Dynamics Reasoning (LDR), reframes the problem: do not regress the next frame directly. Instead, write the state transition as explicit kinematic integration in a structured latent space, and let the network learn only the high-order residual.
It has three stages. Encoding: each frame passes through a CNN to a feature map, then a marginal soft-argmax compresses each channel into a geometric coordinate (centroid μ and extent σ). This structured latent (SL) drops appearance and semantics, keeping only geometry, which makes the later differentiation and integration more stable. Rollout: from three conditioning frames, finite differences give the zeroth-, first-, and second-order latents (analogous to position, velocity, acceleration). The network fθ (a tanh plus an MLP) regresses only the third- and higher-order residual, while the lower orders are integrated by fixed formulas. Decoding: the predicted latent is treated as a warping flow applied to the conditioning frame, twisting out the next frame rather than generating pixels from scratch.
Why does this extrapolate? The authors cite a prior result: outside the training distribution, a network behaves according to its architecture's inductive bias, not its training data. A plain regressor with no dynamics bias collapses to the training mean off-support. LDR bakes kinematic integration into the architecture, forcing the model to learn how the state evolves rather than what the next state looks like, which is what buys extrapolation.
The whole model produces all frames in a single forward pass, with no diffusion sampling and no test-time optimization.
Averaged over the five tasks at 256² resolution, the ID-OOD gap in position error, single-task training:
| Method | ID error | OOD error | ID-OOD gap |
| DiT-S baseline | 0.069 | 0.369 | 0.300 |
| LDR | 0.044 | 0.057 | 0.013 |
Under joint five-task training:
| Method | ID error | OOD error | ID-OOD gap |
| DiT-S baseline | 0.086 | 0.592 | 0.506 |
| LDR | 0.050 | 0.068 | 0.018 |
The baseline is fine in-distribution but explodes out-of-distribution, jumping from 0.086 to 0.592 under joint training, while LDR keeps OOD close to ID. LDR's gap is 1/23.9 and 1/27.7 of the baseline's, roughly a twentieth to a twenty-seventh.
The efficiency gap is even wider:
| DiT-S baseline | LDR | |
| Params | 106.1M | 4.1M (25.9× fewer) |
| 256² latency | 5.21s | 0.036s (143× faster) |
DiT-S runs 50 DDIM steps, each a full transformer forward pass whose cost grows quadratically with token count; LDR emits all frames in one pass.
Both components are necessary. Removing dynamics reasoning (regressing the next latent directly) hurts most: under joint training it even breaks in-distribution, since the model cannot tell the five tasks apart and applies one task's dynamics to another (a downward pull on horizontal uniform motion), landing an ID position error of 0.494 versus LDR's 0.050. Removing the structured latent (using dense convolutional features instead) widens the gap to 0.133.
Higher resolution widens LDR's lead. From 128² to 256², DiT-S's OOD position error grows from 0.222 to 0.592, while LDR's shrinks from 0.114 to 0.068. The two methods scale in opposite directions: more capacity makes the regressor fit the training distribution harder and fail worse out-of-distribution, while LDR reads more precise dynamics from higher-resolution frames.
In stress tests, LDR trained only on red balls still correctly predicts the motion of a Pikachu, a blue square, and reversed-direction objects, where the baseline fails entirely.
This paper gives the overused term "video world model" a testable definition: can it extrapolate the dynamics it learned. It shows that extrapolation comes from kinematic inductive bias written into the architecture, not from a bigger model or more data, consistent with the PhyWorld finding that scaling neither helps a model extrapolate dynamics.
For anyone working on physics simulation, robotic world models, or embodied AI, this is a far cheaper substrate: 4M parameters and a single forward pass beat the diffusion baseline by 26× in size and 143× in speed on controlled physics tasks, and the advantage grows as you push further out-of-distribution. But its current form is far from real-world scenes (see below), making it closer to a proof of principle than a ready-to-use tool.
The authors draw three lines themselves. First, the dynamics reasoning is content-agnostic: kinematic integration assumes nothing about what the latent encodes or which laws govern the pixels, so it does not actively learn physics on its own. Second, the structured latent encodes structure but not content, so it cannot capture dynamics that live in appearance, such as color evolving over time. Third, a geometric coordinate handles simple scenes but may not be expressive enough for richer ones.
The bigger caveat is validation scope. All five tasks are simple rigid-body motions (balls) in a simulator, with single objects and clean laws. Whether this structured latent still holds for real-world video, multi-object interaction, and more complex physics is unanswered, and the authors flag it as future work. Every result is a single run at seed 42, with no error bars.
The "first video world model to extrapolate dynamics" claim needs a discount. It targets the PhyWorld controlled physics benchmark; in the broader video-generation community, next-latent-prediction methods like JEPA move in a similar direction without explicitly modeling dynamics.