MiniWorld: Democratizing the Training of Video World Models from Scratch
Yian Zhao, Ruochong Zheng, Hongcan Guo, Yu Yan, Jian Zhang, Jie Chen
cs.CV
2026-08-02
A from-scratch streaming video world model trained on one 8-GPU server in days; beats a bidirectional short-video baseline by 2x or more on DROID, with code and weights.
A video world model predicts future observations from past frames plus a control signal, the substrate for embodied AI and interactive simulation. The distinction from ordinary video generation is that it has to learn the dynamics governing how an environment evolves under agent actions, not just appearance and motion.
The dominant recipe so far takes a pretrained bidirectional video diffusion model and post-trains or distills it. Bidirectional pretraining lets the model see the whole sequence; streaming inference is causal and cannot look ahead, so the two are structurally mismatched. The pipeline is also heavy and multi-stage. Training autoregressive video world models from scratch has been shown feasible and scalable, but the field has lacked a lightweight, transparent, fully reproducible baseline trainable end-to-end on modest hardware.
MiniWorld trains a block-causal Video Diffusion Transformer with Flow Matching in the latent space of a pretrained Video VAE. Block-causal means tokens within a chunk attend bidirectionally to each other while cross-chunk attention is strictly causal, preserving rich temporal interaction inside a chunk and enforcing autoregressive dependency across chunks.
It builds on Diffusion Forcing with a chunk-wise non-decreasing noise schedule, so earlier chunks are always cleaner than later ones, which matches real inference states. The authors extend Frame-oriented Probability Propagation to the chunk level (CoPP) to keep sampling balanced.
Training is two-stage: short 21/46-frame clips first to learn local action-conditioned dynamics, then 125/253-frame sequences for longer causal context. At inference, a rolling KV cache plus pipelined asynchronous denoising commits finished chunks to a FIFO cache while future chunks denoise concurrently; RoPE re-shifting keeps query positions within the range seen during training.
On DROID and RealEstate10K, against a bidirectional short-video baseline normalized to 1 (relative gains):
| Metric | DROID | RealEstate10K |
| Trajectory accuracy / Photometric smoothness | +249% | +89% |
| Depth accuracy | +238% | +55% |
| LPIPS | +216% | +27% |
| SSIM | +125% | +19% |
| PSNR | n/a | +34% |
| VLM judge score | +63% to 78% | n/a |
On throughput, replacing the 32-chunk full window with an 8-chunk in-flight window plus KV cache lifts steady output from 3.31 to 7.29 FPS (about +220%) and cuts first-chunk latency from 74.0s to 4.86s, a 15.2x reduction. Ablations show that shrinking the denoising window from 32 to 8 chunks and the KV cache from 24 to 6 chunks barely moves quality, with appearance and dynamics metrics within about 1%. The model is MiniWorld-1B, trainable on a single 8-GPU server in a few days.
MiniWorld is valuable as a baseline rather than a leaderboard entry: a from-scratch streaming video world model that is causal throughout, train-test aligned, and cheap enough (one 8-GPU box, days) that groups without giant pretraining budgets can work on video world models instead of adapting expensive bidirectional ones. The results also show that from-scratch training, without piling on compute, can beat the bidirectional short-video baseline by a wide margin.
The authors are explicit that this is a baseline, not a performance ceiling. Model and data scale are small relative to frontier video foundation models, and the evaluated domains cover only a limited range of world dynamics. Long-horizon drift is substantially reduced but prediction errors still accumulate over extended rollouts, especially in complex interactive scenes. How far from-scratch training can be pushed, and whether this cost-effectiveness holds at larger scale, the paper does not answer.