World in World: Explore the World with World Models
Chenxi Song, Yanming Yang, Chi Zhang
cs.CV
2026-09-10
World in World feeds camera-timed visual evidence into a frozen video model, leading VBench overall at 85.2 on DAVIS and OpenVid with lower camera error than six trained baselines.
Autoregressive video world models already support walk-and-look rollouts: change the camera, keep generating. Flexible control is harder. Re-observing a source video along a new camera path requires the rollout to stay synced with the recorded event, place seen content in the requested view, complete newly exposed surfaces, and recover appearance on revisits. Prior systems usually add a module per control and train it. A new evidence type means a new pathway.
World in World (WiW) treats this as an inference-time interface. Heterogeneous evidence is converted into camera- and time-labelled clean visual states. A frozen backbone reads them through native self-attention.
The method is instantiated on the public causal-fast checkpoint of LingBot-World 2.0, with all pretrained weights frozen. Every evidence channel is converted the same way: visual content, per-frame camera and event time, token-level spatial support, and a flag for which denoising steps the channel is active. Clean states are passed through the backbone at diffusion timestep t=0. Layer-wise K/V are cached. Queries from the current chunk read them.
Four sources do different jobs:
Where to read is correspondence-guided attention routing (CGAR): persistent point identities plus geometry bias current queries toward matching source-video tokens. How hard to read is evidence-wise attention CFG (EWA): in the same denoising forward pass it compares the native-block response with the response that includes one evidence channel, and amplifies only the component orthogonal to the native direction. No extra network evaluations are added for guidance.
The main task is camera-controlled video rerendering on DAVIS and OpenVid-1M. Baselines are ReCamMaster, TrajectoryCrafter, WorldForge, InSpatio-World, UniWorld-View, and CameraAnything, each with its official code and recommended frame count.
| Method | VBench Overall | RotError (°) | TransError | PSNR |
| ReCamMaster | 83.836 | 7.86 | 0.124 | 15.14 |
| UniWorld-View | 84.295 | 4.20 | 0.069 | 22.47 |
| CameraAnything | 83.036 | 3.19 | 0.186 | 15.16 |
| WiW | 85.192 | 2.83 | 0.069 | 23.15 |
WiW leads the seven-dimension VBench average, rotation error, PSNR, and SSIM. Removing target-view warping hurts most: rotation error rises about 3.4x and translation error about 10.9x (RotError 6.12 vs 1.78, TransError 0.58 vs 0.053). Dropping CGAR or EWA barely moves camera error and mainly hurts subject and background consistency. The same interface also runs bullet-time, stabilization, editing, K/V sharing between two generation cases, and human-motion transfer.
For anyone sitting on a frozen video world model, this is a control extension that adds no modules and no fine-tuning. Control becomes evidence construction, alignment, and weighting. Camera rerendering is the headline eval; the interface is not tied to that task.
The VBench Overall gap versus UniWorld-View is under one point. The separation is rotation error, plus the training-free constraint. If the backbone cannot be updated, this is the more usable path. If a dedicated control head can be trained, the paper does not show that WiW beats that ceiling.
Evaluation concentrates on camera-controlled rerendering. Long-horizon revisits and motion transfer are mostly qualitative. The geometry channel depends on a human avatar and SMPL-X; how non-human subjects complete newly exposed surfaces is specified as an interface, not measured at the same scale. Depth comes from DepthCrafter, so projection error enters attention.
WiW is not first on every VBench axis: Temporal Flickering is 93.711, below ReCamMaster's 95.178. Camera error is estimated from generated video with Depth Anything 3 and ViPE, then compared with the target path, so estimator error enters the metric. Frame counts follow each baseline's official setting and are aligned after the fact, which is not a fully matched protocol. Latency and memory are unreported, so the cost of temporary auxiliary attention blocks is unknown.