Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training
Wenxuan Shen, Dongna Jin, Dongping Chen
cs.CV
2026-08-25
G2WEngine builds 96K pairs from 5,132 HUD assets in 303 games. Mask-free GameCleaner hits 95.36 synthetic AAR and 80.05 wild AAR; cleaned clips lift Wan VideoReward 6.83%.
Internet gameplay looks like free world-model data: many scenes, messy interactions, long trajectories. What you actually download is a composite frame. Health bars, minimaps, quest toasts, kill feeds, facecams, and watermarks sit in screen space. They are not part of the game world, and they still enter the training signal.
The paper isolates that confound. It takes 5,442 clean clips, keeps one copy untouched, paints synthetic HUDs on the other with G2WEngine, and finetunes Wan-2.1-T2V-1.3B on each split. Clean data raises overall VideoReward of generated video by 6.83%, and motion quality by 18.8%. Clarity drops 1.43%, which is expected: sharp HUD text inflates sharpness metrics.
The bottleneck is not footage volume. Interface and world are tangled, and the paper treats disentangling them as a data pipeline.
G2WEngine runs in four stages.
Game2World-S has 96K such pairs. Game2World-W has 1,079 real 5-second clips from the same 303 games and no clean target. GPT-5.6 Sol labels the first frame; humans patch dynamic widgets on later frames.
GameCleaner removes UI without an input mask. The architecture follows Kiwi-Edit: an MLLM encodes sampled source frames plus a removal instruction, learnable queries pull out a condition for a video DiT, and source VAE latents re-enter through a timestep-dependent residual so the scene does not drift. The MLLM and connector stay frozen. Only rank-64 LoRA on the DiT is trained, 3,000 flow-matching steps, 8xH100, global batch 32, learning rate 1e-4, 720p. A reference-conditioned variant drops the clean reference on 20% of steps so it can run with or without a clean frame at test time.
After finetuning, clean-data VideoReward is VQ -0.643, MQ -0.298, TA 1.008, overall 0.068; the UI-overlaid twin is -0.661, -0.367, 1.018, -0.009. Aesthetic scores 6.109 vs 5.626; motion 16.392 vs 15.706.
Removal is judged by an MLLM at 2 FPS. On 200 videos and 6,150 element labels the judge hits 87.78% removed F1 and Cohen's κ of 0.820. AAR discounts artifacts: 1.0 for a clean removal, 0.5 with minor artifacts, 0 if the widget remains. BG scores whether pixels outside the boxes stay put.
| Method | Syn AAR / BG | Wild AAR / BG | Overall |
| VACE-1.3B (mask) | 59.11 / 96.50 | 30.40 / 91.45 | 69.37 |
| EffectErase (mask) | 56.17 / 84.45 | 23.55 / 74.55 | 59.68 |
| Aurora (mask-free) | 60.62 / 24.45 | 51.19 / 14.00 | 37.57 |
| Kiwi-Edit (mask-free) | 34.68 / 97.30 | 13.54 / 96.80 | 60.58 |
| GameCleaner | 95.36 / 99.00 | 48.56 / 99.60 | 85.63 |
| GameCleaner (w. ref) | 94.64 / 98.85 | 80.05 / 99.80 | 93.34 |
On synthetic AAR, GameCleaner is 57.3% above Aurora, the strongest mask-free baseline. EffectErase's raw "removed" rates look strong (94.56 synthetic, 92.00 wild), then AAR and BG collapse, because nearby scene content moves. With a reference, GameCleaner Track-Clean on wild video is 68.12 against EffectErase's 16.12.
A 20% reference drop is the useful setting: wild AAR goes from 70.64 at 0% drop to 80.05, then falls to 56.23 / 54.64 at 50% / 80%. Scaling data from 60% to 100% adds only 2.06 synthetic AAR and 25.09 wild AAR. Synthetic performance is near a plateau; wild performance still wants diversity.
Genie, GameGen-X, and MineWorld mostly treat rendered frames as world observations. Public livestreams dwarf instrumented simulators, and HUDs have been the tax on that footage. This paper turns HUD stripping into a clip-level, parallel data job rather than a per-game editing script.
For anyone training on gameplay video, the operational point is narrow: take the interface off first, or motion and aesthetics will move for the wrong reason. Ranking models by raw "is the UI gone" overrates editors that rewrite the background. GameCleaner does not need a mask and is tested across 303 layouts. Code, data, and weights are promised on GitHub.
This is data engineering, not a new world-model architecture. Downstream checks still use text-to-video as a proxy.
The authors are explicit. Game2World does not cover the full spread of games, UIs, and capture conditions. Large opaque panels, fast flashing overlays, and widgets that share texture with the scene reconstruct poorly. There is no action conditioning and no cross-game inverse dynamics, so cleaned video is still observation-only. GameCleaner runs a full diffusion trajectory; few-step distillation is left for later. They are expanding the set toward 1M pairs.
A few claims sit uneasily. The introduction cites a synthetic average of 0.5697 and a 40.3% gap over LoomVideo, which does not match Table 3 (LoomVideo overall 42.82). Trust the table. The 6.83% VideoReward lift is about T2V supervision, not an interactive world model. Judge scores shift with evaluation resolution from 448p to 1080p, so comparisons have to lock resolution. Synthetic overlay-and-peel is not the same as real semi-transparent HUD, motion blur, and compression; wild AAR falling from 95 to 48 without a reference is the actual generalization gap.