Building Pretraining Data for World Models: An Unreal Engine-Based Pipeline for Action-Conditioned Video Generation
Haoyu Wang, Songchun Zhang, Haoran Li, Haoyang Huang, Zeyue Xue, Nan Duan
cs.CV, cs.GR
2026-09-03
A two-stage Unreal pipeline on 25 nodes of eight RTX 5090s turns 2,384 Fab packs into 429 levels and 8,767 hours of five-view, action-aligned video for EchoWM.
Interactive world models need more than pretty video. They need control signals temporally aligned with the pixels. Internet video is visually broad, but the actions that caused each transition are usually unknown. Gameplay traces have buttons, yet they are tied to one title. Systems such as Genie, GameNGen, Matrix-Game 3.0, DreamX-World, and EchoWM are pushing resolution, horizon, and controllability, and their data engines now mix synthetic 3D, gameplay, and web video. The synthetic slice is often a paragraph in a model paper.
This report is that slice for EchoWM. The job is to turn heterogeneous third-party Unreal assets into multi-view video whose actions, poses, and cameras are recorded from the engine, not inferred after the fact.
The engine itself forces the design. Physically valid motion needs a live tick so collisions, gravity, and slopes resolve. High-quality frames need Movie Render Queue, which evaluates a timeline offline and may sample a single output frame many times. Capturing with SceneCapture during the physics run never produced stable anti-aliasing in their scripted setup. Simulation and beauty passes cannot share a process.
The two stages are a contract with the engine, not a modeling flourish.
The action set has nine discrete states: WASD, the four diagonals, and IDLE. A standard trajectory is 1,800 frames (60 s) with five synced cameras: back follow, left, right, top-down, and a near-head first person. External cameras use character-relative offsets. A ray from the character to the desired camera pulls the camera inward on occlusion, contracting fast and expanding slowly so thin props do not pop. Actions are sampled from a fixed distribution and held for a random run of frames to avoid jitter. Facing is biased toward points of interest without changing the action probabilities. Dead-ends trigger a large rotation, then a respawn.
Assets come from Fab. Of 2,384 packs, the scanner drops incomplete sub-levels, asset-preview layouts, and partitioned open worlds. Renderability is checked on five 10-second 540p clips: crashes, no walkable region, hovering, or heavy interpenetration disable the level. Visual filters reuse SpatialVID's aesthetic and luminance scores. A view passes if aes>5.5 and 25<lum<140; a level enters production only if more than 25% of sampled clips pass. Characters get their own calibration: nine animations, collision box, yaw correction, and walk/fly/swim mode.
The cluster is 25 nodes, eight RTX 5090s each. A central allocator hands out level pools; each node owns slot lifecycle. Levels are partitioned persistently so shader and texture caches survive. A cold scene can take more than ten minutes; later visits drop below half that. Slots oversubscribe GPUs because processes spend long stretches loading and compiling. Progress is an append-only log. After a crash, only trajectories with both a completion marker and complete videos are kept. Upload is asynchronous. Failure snapshots ship logs off-node before cleanup.
This is a production report. There is no downstream world-model bake-off in these pages.
| Item | Value |
| Fab packs / levels kept | 2,384 / 429 |
| Humanoid characters / camera views | 40 / 5 |
| 1080p / 720p output | 2,691 h / 6,076 h |
| Five-view throughput | 33 min per node-hour |
| Diagonal + back + lateral share | 46.6% |
Historical/fantasy, sci-fi, nature, and urban/architecture each take roughly 19–29% of production scenes. Aesthetic and luminance cutoffs were chosen by eye. They have not been calibrated against world-model training.
The world-model race is as much about whether action, pixels, and cameras actually line up as it is about another generator. This write-up pulls synthetic data out of the model appendix and documents the operational contract: why physics and MRQ must split, how cache-aware pools beat uniform sampling, how cinematic actors and texture streaming get stripped from third-party maps. Anyone building an interactive world-model or embodied data engine can copy the two-stage handoff instead of rediscovering that SceneCapture anti-aliasing will not hold.
It does not show how many FID or control-error points these hours buy. EchoWM is the paper that trains on them. Read this to learn how 8,700 hours were made, and which filters are still unanchored to training metrics.
The authors are blunt. Controllability, action diversity, physical plausibility, image quality, and throughput pull in different directions. Aesthetic and luminance scores remove black frames, blowouts, and empty shots; they are not a proxy for useful dynamics, and over-filtering can drop structurally rich ugly scenes. The action set is locomotion plus idle, with no jump, run, crawl, attack, or climbing. A newer variant is moving toward camera-centric controls (forward means deeper into the image); this report is still mostly character-centric. Third-party collision and cinematic cameras still need a human look at 540p clips. That gate is not fully automatic.