From Generation to Simulation: How Far Are World Models from Being True Simulators?
Tong Wang, Huan Deng, Mucheng Yang, Yang He, Xiaohui Kuang, Gang Zhao
cs.AI, cs.CV
2026-08-24
A 200-paper audit maps world models to eight simulator capabilities. Interaction and control can substitute in some settings; only 6 of 163 expose queryable entity state.
Video diffusion and world foundation models have turned "world model as simulator" into something you can demo. GameNGen runs around 20 FPS in a constrained game domain; the Matrix-Game series reports 720p at about 40 FPS. The 2018 VAE-plus-RNN imagination setup from Ha and Schmidhuber has been pushed, via Dreamer, Genie, the Sora debate, and Cosmos, into driving, manipulation, and open-ended exploration. Existing surveys still mostly classify world models by architecture, function, or application. That does not measure the remaining distance to a physics engine, a game engine, or an RL environment.
This paper swaps in an external ruler: eight capabilities of a traditional simulator. The scale does not belong to any one technical route, so latent dynamics, video generation, and JEPA can be plotted on one map.
Sources are Google Scholar, arXiv, DBLP, and Crossref, cutoff 30 June 2026, expanded along six surveys and the citation graphs of World Models, DreamerV3, Genie, and Cosmos. Inclusion requires an action or conditional input and a forward rollout of future state. Pure text-to-video backbones, representation learning without action conditioning, trajectory-only predictors, and papers that say "world model" but do language encoding or cognitive modelling are out. The corpus is 200 papers: 163 implementations, 18 benchmarks, 16 surveys. 72 are formally published; 128 remain preprints.
Implementations split into six families: latent dynamics / latent action (29), autoregressive (37), diffusion (63), JEPA (5), explicit 3D/4D (25), occupancy (4). Each paper is tagged with one to three principal-contribution capabilities. All 163 implementations also get a five-variable state-interface audit: B1 ego information, B2 sensor output, B3 reward and termination, B4 runtime-queryable non-ego entity or physics parameters, B5 closed-loop interaction. Present, absent, and unresolved are counted separately, so missing evidence is not treated as absence.
Counted as a principal contribution, research attention is badly skewed. Controllability appears in 125 papers (62.5%); interaction and stability in 80 each (40%). Asset construction has 38 papers (19%), the physics engine about 17%, state feedback 45 (22.5%). The authors tag the first three as relative strengths and the last three as cross-route structural gaps.
The state-interface audit is the hard number. B1 is confirmed in 65 papers, B2 in 45, B3 in 20, B5 in 87. B4, a runtime interface for named entity or physics fields, is present in 6, absent in 157, unresolved in 0. Sensor outputs beyond RGB are explicitly reported in 45 of 163. In GameNGen's human study, only 58–60% of people could tell real Doom frames from generated ones. The pixels can pass as a simulator. The query interface cannot.
V-JEPA 2-AC cuts planning from about 4 minutes in pixel space to about 16 seconds, roughly 15×, at the cost of an unreadable latent. Route mixing is already happening: causal-forcing distillation, few-step autoregressive diffusion, explicit 3D memory plus implicit appearance, VLA fused with a world model. The judgement is blunt. In restricted settings, interaction and controllability can substitute. Formal physical guarantees, structured state feedback, and reproducible long-horizon evolution are still a step short.
If the plan is to replace MuJoCo, CARLA, or Unity with a world model for embodied, driving, or game RL, this audit separates "looks right" from "plugs in". Policies, closed-loop eval, and safety analysis need queryable entity state, not a better FVD. The six follow-ups map onto those gaps: verifiable physics, a unified action interface, state feedback as a first-class output, long-horizon stability, downstream success instead of generation quality, and cross-route fusion. World-in-World has already reported that visual quality is not task success. The agenda is not empty.
Capability tags measure research attention, not attainment. Physics that is present but not a principal contribution does not land in the physics column. JEPA has 5 papers and occupancy 4, too few for family rankings. Dropping pure video generation biases the corpus toward action-conditioned work. 128 of 200 records are still preprints; several fields depend on author self-report, and B1 is unresolved in 24 papers. The cutoff is June 2026. There is no head-to-head error curve against a traditional simulator; the gap is a structural inventory, not a matched benchmark.