WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations
Xiaojie Xu, Zhengyuan Lin, Runyi Li, Yihao Liu, Kaipeng Zhang, Yongtao Ge
cs.CV, cs.GR
2026-08-16
Alaya Lab, University of Tokyo and Shanghai Innovation Institute build an Unreal Engine offline-rendering pipeline that walks artist-built environments along minute-scale routes and replays each from three camera systems under different weather, yielding 21.9M frames with metric depth, optical flow, long-range tracks and trajectory-derived action signals.
Training world models (systems that generate or reconstruct explorable environments) needs supervision beyond RGB: camera pose, metric depth, long-range correspondence, and for interactive models, control signals. In real footage only the pixels are ground truth; pose is estimated after capture and drifts, and correspondence through occlusion fails when matched algorithmically. Rendering hands these quantities over directly, but existing synthetic resources each miss something: OmniX ships multi-view video with no action stream, WildWorld records gameplay with a 450-plus action vocabulary but no control over scene or lighting, and few datasets put depth, flow, tracks and actions on the same frames while also supporting viewpoint and appearance swaps.
WorldRover's move is to define an exploration as a reusable world-space trajectory rather than a finalized video. Fix the trajectory, and you can re-observe it from any viewpoint under any appearance state.
The engine sits on Unreal Engine with an editor side and a render side:
The key capability is replay: one route can be rendered from a first-person eye-height camera, a third-person setup where an animated character walks the path with a follow camera several metres behind, and a 360-degree panoramic mode rendering six 100-degree cube faces at the same instant into a 4096×2048 equirectangular video, or under a different weather state, or with every surface swapped to a neutral white material. Route, timing and scene geometry are preserved throughout, so the variants are pixel-aligned contrastive data.
Annotations come as renderer ground truth: metric depth (0.1 to 200 m, log-quantized 16-bit), per-frame camera intrinsics and extrinsics, dense optical flow from the renderer's velocity buffer, and long-range 2D/3D point tracks obtained by geometric projection rather than matching, with visibility flags. Third-person sequences add a character trajectory that genuinely differs from the camera's: the character turns while the follow camera holds its azimuth. The action stream is a control representation reconstructed from the motion: velocity differences between adjacent frames resolved into forward, lateral, turning and vertical axes in the first-person camera frame, quantized, written on change.
The current release, WorldRover-10M (the name refers to its 10.8M first-person frames):
| Viewpoint | Sequences | Frames | Duration | Size |
| First-person | 2,910 | 10.8M | 100.5h | 5.9TB |
| Third-person | 1,885 | 8.3M | 76.5h | 4.4TB |
| 360 panoramic | 1,208 | 2.8M | 25.8h | 8.4TB |
| Total | 6,003 | 21.9M | 202.7h | 18.65TB |
Built from 32 environments and more than 70 animated characters (humanoids, quadrupeds, birds). Standard viewpoints are 1280×720; panoramic is 4096×2048. Median sequence duration is 115 s for planner routes, 105 s for waypoint, 81 s for reactive. Urban scenes account for 2,128 sequences and period towns 1,668, the two largest blocks.
There are no downstream training experiments. The authors state this plainly: the report describes the pipeline and the data, and the next step is to train on the released data and measure on standard benchmarks.
For anyone working on world models, video generation, depth, optical flow or point tracking, this is a training source with the supervision signals bundled and geometrically consistent. The cross-viewpoint, cross-weather replays of one route form natural contrastive pairs: the same world under different observations should agree, which is exactly the constraint world consistency is made of. The white-material variant hands you coarse-to-fine structural control for free, and the four-axis action stream matches the control interface of interactive world models that take forward-and-turn commands.
The honest caveat: this is a dataset report with zero downstream numbers proving the frames train better models. Its value rides on future training results, and it is too early to call.
Three stated by the authors, all substantive:
One more from my reading: the action stream is reconstructed from trajectories, not logged from actual input events. The distributional gap between reconstructed actions and what a player would actually do is not discussed, which matters if you use it for action-conditioned generation.