SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models
Junchao Huang, Guian Fang, Shengju Qian, Xianghao Kong, Zhuoran Zhao, Wei Huang, Yihua Du, Zixin Zhang, Justin Cui, Yuchao Gu, Yukang Chen, Xinting Hu, Tianyu He, Shaoshuai Shi, Zhuotao Tian, Xin Wang, Mike Zheng Shou, Li Jiang
cs.CV
2026-09-03
SolarWM canonicalizes 1.43M clips from 10 sources and adapts Wan, LTX, and MiniMax with one three-stage recipe into interactive world models.
Interactive video world models roll future frames under camera motion or instructions, turning a generator into an explorable environment. Short-clip generation already looks plausible. Long-horizon interaction still breaks in two coupled places. Datasets disagree on duration, camera conventions, quality, and captions, so naive mixing yields inconsistent supervision. Video backbones disagree on latents, attention, and conditioning, so a custom training stack per model makes results incomparable.
Open systems often ship weights or inference only, without processed data, selection records, or exact recipes. SolarWM aims to be the missing reproducible foundation: a data engine plus backbone-native adaptation.
The data engine converts 10 sources (ABOT-World, DL3DV, MiraData, RealCam, SpatialVID, Sekai, MIND, MultiCamVideo, OmniWorld, and related views) into about 1.43 million canonical clips, 25.85 TB on disk. Each sample follows one frame-aligned contract: video, metric camera-to-world poses, per-frame intrinsics, dense captions, quality metrics, and provenance. Source processing is decoupled from mixture construction, so changing filters or source weights does not rerun camera estimation or captioning.
Camera annotation is source-aware. Video-only clips go through Pi3X, MoGe-2, and a VIPE SLAM backend. Sources with COLMAP or ground-truth poses keep the trajectory and only align scale. Dynamic people and vehicles confuse camera-controlled training, so LTX-2.3 Clean Plate produces three extra owners totaling about 543k clips. Captions come from one Kimi-K2.6 pipeline that describes the stable environment and omits camera motion, so text does not leak the control signal.
Under shared camera, training, and rollout interfaces, the paper instantiates four backbones: Wan2.2 5B/14B, LTX-2.5 22B, and MiniMax-H3 33B. Camera control uses fused-PRoPE on existing Q/K/V, with no extra control branch. All three training stages run on 5-second sequences:
Inference is 16 fps, four sampling steps, no attention sink.
The experimental section is almost entirely qualitative. After bidirectional pretraining, all four routes generate 10-second clips from out-of-domain first frames (GPT Image 2 or Krea) under translation, rotation, and mixed 6-DoF paths. The distilled Wan2.2-5B-fast causal student then autoregresses 10-second and minute-scale in-domain scenes from OmniWorld, MiraData, Sekai-Game, MIND, ABOT, and DL3DV; minute-scale out-of-domain rollouts from synthetic stills; and uninterrupted hour-scale runs from real first frames. Training used 5-second windows. Evaluation does not splice short clips, inject reference frames, or use an attention sink.
Comparison with other systems is a release matrix, not a FID or camera-error table. SolarWM commits to weights, inference, training, 25.85 TB / 1.426M clips, full selection records, an executable pipeline, exact recipes, and four backbone families. SANA-WM is listed at about 235 GB / 1.6k clips, minWM at about 527 GB / 19k clips. Most peers omit training code or data recipes.
The scarce resource in world-model research is a stack others can reproduce and reconfigure. Decoupling preprocessing from mixing lets later work change filters without rerunning 25 TB of annotation. The recipe is intentionally short: the authors argue compute should stay in the bidirectional stage, with rapid AR adaptation and even shorter DMD, and without long-video fine-tuning.
For camera-controlled walkthroughs, games, or embodied simulation, this is infrastructure. It is not a scored bake-off.
The main experiments report no FVD, camera-trajectory error, user study, or matched numeric comparison against Genie, Yume, or AlayaWorld, so the "state-of-the-art" claim has no leaderboard to check. Hour-scale results are sparse-frame galleries; drift, texture collapse, and object persistence lack a quantitative protocol. Clean Plate can add artifacts. Captions strip dynamic entities, which hurts scenes that need people or traffic. Control is mostly camera motion, not keyboard or robot actions. Release language is "we will release"; readers should verify what actually landed.