WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory
Wangbo Yu, Kunhao Liu, Wenbo Hu, Shenghai Yuan, Chaoran Feng, Haiyang Zhou, Yukun Huang, Yiran Wang, Wang Zhao, Yingmin Luo, Ying Shan
cs.CV, cs.AI, cs.GR
2026-09-22
WorldCrafter compresses history into a pose-queryable implicit 3D memory. Revisit LPIPS falls from Lyra 2.0's 0.487 to 0.255; the distilled model streams at 16 fps on four GPUs.
An interactive video world model has to follow a user-driven camera and still recognize a room after minutes of wandering. Full-history attention does not scale. Retrieving a few similar frames covers little of the scene. Warping old frames with estimated depth breaks when geometry is wrong or objects move. Geometry-pretrained implicit memories lean toward depth, and do not keep appearance the way novel-view synthesis does.
WorldCrafter wants a camera-queryable implicit 3D memory: history is compressed into a fixed token budget, and the upcoming viewpoint decides what gets packed.
The backbone is a chunk-wise autoregressive video DiT, initialized from Helios-base. The first chunk is generated from camera poses alone. Later chunks see three streams: memory tokens, a short recent context, and the target camera trajectory.
Memory writing: the encoder starts from LagerNVS, drops the shallow image stack, and reads VAE latents directly. Encoder input is capped. At inference the latest latent is kept, and the remaining history frames are chosen greedily so their fields of view jointly cover the upcoming target region (max-coverage), rather than ranking frames one-by-one by pairwise FoV similarity.
Memory readout: under a fixed DiT token budget the paper compares pose-free readout with pose-guided readout. The latter queries the memory with poses sampled from the upcoming trajectory, so the budget is spent on what the camera is about to see. Those tokens, recent history, and the current noise form one self-attention sequence. Camera conditioning uses a parallel UCPE attention branch, injected only into the noisy segment, not into memory or recent context.
Training has four stages: adapt the inference window, freeze the backbone and train the camera branch, warm the encoder on latent input, then jointly train encoder, readout, DiT, and camera branch. Data mixes Open-Sora-Plan, DL3DV, and MIND synthetic video, with poses from Depth Anything 3. Distillation uses pyramid DMD with two students: a low-noise model trained for texture, a high-noise model trained for subject following. High noise runs the early steps, low noise the last. WorldCrafter-fast reaches 16 fps on four GPUs.
The benchmark has 145 images (83 dynamic, 62 static), each paired with five closed-loop trajectories, for 725 videos of 528 to 1,648 frames. Baselines include DreamX-World, Alaya-EVOKE, HY-WorldPlay, Lyra 2.0, Echo-WM, LingBot-World 2, Matrix-Game 3.5, and SANA-WM.
Revisit consistency:
| Method | LPIPS↓ | PSNR↑ | MEt3R↓ |
| Lyra 2.0 | 0.487 | 14.050 | 0.334 |
| WorldCrafter | 0.255 | 18.016 | 0.166 |
| WorldCrafter-fast | 0.186 | 20.868 | 0.129 |
Relative to Lyra 2.0, LPIPS falls 47.6%. Camera control: WorldCrafter has the lowest RotErr 13.536, TransErr 1.475, and CamMC 1.546. The distilled model is weaker on camera (RotErr 18.251) and still ahead of most baselines. VBench overall is 81.910 for WorldCrafter, best on five of eight axes; Alaya-EVOKE sits at 81.406. Dynamic degree is slightly below some game-oriented baselines.
Ablations at the same token budget: context memory (four raw history frames) LPIPS 0.497; frozen encoder 0.305; pose-free readout 0.333; similarity retrieval 0.296; full model 0.255. Joint training drives validation LPIPS down earlier than a frozen encoder. Memory processing is 0.062 s per chunk against 1.346 s for a depth-warp pipeline, about 21.7× faster.
If a world-model stack still estimates depth and warps, this memory is the alternative to try: appearance and geometry compressed together, read out by camera query, no per-frame explicit correspondence. For minute-scale walkthroughs, 16 fps after distillation means the memory path does not have to stall streaming generation.
The work is incremental. It does not claim geometric reconstruction accuracy; point clouds are visualizations. Reappearing subjects are shown qualitatively, without a leave-and-return duration curve.
Consistency still breaks on especially complex or long trajectories. Re-encoding history every chunk adds latency; the authors suggest an incremental streaming encoder. Trajectories are a custom closed-loop set, not each baseline's official protocol, and videos are resized to 640×384. WorldCrafter-fast is stronger on appearance and weaker on camera, a distillation tradeoff. How much of the gain is LagerNVS's 3D bias is unclear: there is no random-init encoder ablation.