GameWAM: A World Action Model for Video Games
Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Weijia Li
cs.AI, cs.CV, cs.LG
2026-08-26
GameWAM jointly generates future frames and keyboard-mouse actions. It reaches 50.7% MCU Mini success versus OpenHA’s 36.8%, using 138 native steps per successful embodied episode.
Game agents map pixels and a task prompt straight to keyboard-mouse commands. They do not model how the world will look after those commands fire. Interactive game world models do the opposite: they roll future frames given actions, and they are not task policies. World-Action Models generate both the future observation and the executable trajectory. Prior WAMs mostly sit in tabletop or bounded indoor manipulation. Native closed-loop play, with first-person motion, persistent state, and mixed keyboard-mouse control, had not been tried.
Minecraft and ViZDoom make the gap concrete. The same physical axes mean camera yaw in gameplay and cursor motion in a GUI, with different scales and conditionals. Visual tokens are also subsampled relative to native actions, so denser video buys less look-ahead under a fixed budget.
GameWAM runs two parallel Diffusion Transformers. The Video DiT starts from Wan2.2-TI2V-5B. The Action DiT keeps the same 30-layer, 24-head layout but shrinks hidden width from 3072 to 1024, with weights adapted from the video backbone. Both streams train with joint flow matching. Attention is block-causal: noisy video and noisy actions share a clean visual prefix and do not condition on each other. At test time the policy can emit actions without denoising future video.
A timestep-wise router picks gameplay versus GUI, then the matching prediction head and continuous-action normalization. Discrete keys share one scale. Invalid channels are masked.
Long-horizon control uses a block-cycle: predict P=16 native actions, execute the first E=8, observe, replan. Three execution blocks make a 24-action cycle. The in-cycle KV cache is dropped at the boundary. Across cycles the model keeps the two most recent compressed segments and older content in attention-pooled slots, 40 tokens at most. Training teacher-forces overlapping P-step plans on an E-spaced grid.
Three aligned streams feed Minecraft training: 80% event-anchored VPT, 5% regular VPT, 15% scripted GUI. Sampling is dense near events (stride 2) and sparse away from them (stride 16). Two epochs on eight H200 GPUs take about 22 hours and consume 2.79B tokens on the reported schedule. Inference uses ten first-order steps.
MCU has more than 800 tasks. Mini average success is 50.7% against OpenHA at 36.8%. Full-set average is 46.6% against Game-TARS at 42.5%, the latter marked as multi-game pretrained. Successful episodes use fewer native steps in every category:
| Category | GameWAM steps | OpenHA | Game-TARS | GameWAM ASR All | Game-TARS ASR All |
| Embodied | 138 | 287 | 373 | 47.5% | 50.4% |
| GUI | 155 | 314 | 406 | 60.0% | 39.1% |
| Combat | 203 | 316 | 372 | 32.2% | 38.1% |
The average lead is a GUI lead. Embodied and combat full-set success still trail Game-TARS. Variance is large: Mini embodied is 70.0±32.2.
ViZDoom is four maps, 50 episodes each, plotted as mean reward. GameWAM beats Game-TARS on all four and is competitive with or ahead of o4-mini, GPT-5, Gemini-2.5-Pro, and Claude-4-Sonnet. Exact per-map numbers are not tabulated in the text.
MCU Mini ablations: drop future-video supervision and the average falls from 50.7 to 35.7; coarser temporal sampling 36.7; no event-anchored clip sampling 38.0; a unified gameplay/GUI action distribution 38.3; P=E 41.3; no cross-cycle history 46.7, with embodied rising to 75.0 and GUI falling to 31.0.
LASI is the sharper finding. Reusing one sampled action source across replans can lock the camera into a directional bias, including in-place spinning. Under fixed conditioning, yaw DCT0 correlates with the source low band at r=0.890. Replacing source modes 0–2 makes yaw DCT0 follow the donor in 94.8% of trials. Zeroing that band removes 99.25% of the associated output variance. Resampling the source at every replan stops coherent accumulation. It does not remove the sensitivity. Training-time patches that suppress the coupling tend to hurt action learning.
A cloned key-mouse policy can finish a task without ever predicting the pixels that should follow. GameWAM shows that joint world-action generation can match or beat strong Minecraft agents with fewer executed actions, especially on GUI. For anyone training generative policies, LASI is the part to keep: a flow source can imprint a low-frequency control bias, and closed-loop reuse amplifies it.
GameWAM is still a low-level controller. It does not keep a recipe graph or an inventory plan. It is not the same interface as hierarchical or heavily pretrained agents.
The authors mark the scope themselves. There is no symbolic task graph. Persistent history is mostly visual, and the auxiliary loss only predicts the current clean visual feature. Evaluation stays in digital games. MCU leans on atomic tasks, not long chains of coherent intent. LASI has no mechanism-level fix that keeps task performance.
Game-TARS carries a multi-game pretraining flag and still wins embodied and combat on the full set. Mini variance is wide enough that a point estimate is not a stable ranking. ViZDoom is a figure, not a table.