From Pixels to States: Rethinking Interactive World Models as Game Engines
Zhen Li, Zian Meng, Shuwei Shi, Mingliang Zhai, Jiaming Tan, Chuanhao Li, Kaipeng Zhang
cs.CV
2026-07-16
An analysis paper auditing interactive world models through a game engine's action-state-observation loop across four dimensions, plus a 90-hour frame-aligned Black Myth: Wukong dataset of player actions and game states.
A recent wave of video generative models offers a data-driven path to interactive worlds: predict the next frame conditioned on the player's action, and you have a candidate next-generation game engine. But a genuinely interactive game world needs three things at once: interaction outcomes that follow rules over evolving state, consequences that persist over long horizons, and real-time generation. Conventional engines realize these through an action-state-observation loop: player actions update an explicit game state by predefined rules, and observations are rendered from that state. This paper takes that loop as a lens and audits where current methods fall short.
There is no new model. Across four dimensions, the authors group existing work into representative families and discuss each one's tradeoffs:
They also release a scalable data engine for Black Myth: Wukong: 90 hours of gameplay at 1280x720, 30 FPS, focused on high-interaction boss-encounter rollouts. Frame-aligned per tick: raw mouse and keyboard input, game state (poses, animations, skills, attributes), RGB plus depth, plus structured slot captions and semantic captions generated with Qwen3-VL-235B.
There is no benchmark table of model comparisons. The "result" is a map of the field plus a dataset: the four dimensions make each family's strengths and weaknesses explicit, and the dataset fills a concrete gap, state-aware game world modeling lacks data. The authors' central claim is that existing methods deliver natural input control and near-real-time generation, but the hard capabilities all "revolve around the game state," which most models keep implicit.
For anyone working on world models or game AI, this paper saves time: the four-dimension framework lets you locate your own work in the family tree without rebuilding it from scratch, and the per-family tradeoffs are spelled out. The 90-hour, frame-aligned, state-annotated dataset is the more concrete contribution. Interaction data with ground-truth game state is scarce, and this is raw material for directions that plug explicit state into the generation loop. It is a position paper plus a dataset, not a benchmark chase, so calibrate expectations accordingly.
This is an analysis and position paper, not an empirical study: no model, no reproducible experiments, and every claim about which family is stronger is qualitative, with the grouping itself a subjective choice. The dataset covers a single game (Black Myth: Wukong) and focuses on boss encounters, one high-interaction regime; whether it represents open-world exploration or varied play styles is unclear, and 90 hours is narrow relative to real game diversity. The four-dimension framework is the authors' organizing lens, not an established taxonomy. The paper is meant to foster progress, but a position paper's impact ultimately depends on whether the community adopts the taxonomy and whether the dataset gets used, neither of which is yet visible.