WorldMind: Decoupled Game World Model for State-Aware NPC Behavior
Zhiyang Deng, Boran Zhang, Danze Chen, Yeying Jin
cs.CV
2026-08-18
WorldMind lets NPCs plan from compact state, not pixels. On Game A one-minute rollouts it is preferred in ~70% of pairwise comparisons versus implicit and external NPC control.
Game world models can already render the next frame from player input. Oasis, Genie, and Matrix-Game all sit on that loop. NPC behavior has not caught up. In most systems the boss is a motion pattern absorbed into the video prior. In others the NPC action is an external control channel the world model never chooses.
Neither setup gives the NPC an interface that reads the fight as it evolves. A boss encounter turns on distance, facing, the last skill, and whether cooldowns are up. Packing perception, planning, and rendering into one diffusion pass leaves the NPC slow to react.
WorldMind splits interactive world modeling into four layers and closes them in a loop. The claim is that this is the first game world model to treat state-aware NPC behavior as an explicit decision, separate from pixels. The work comes from Tencent and the National University of Singapore, done during internships.
Understanding (L1) rebuilds a compact state from the frames just generated. A geometry branch takes three causal RGB frames at offsets -4, -1, and 0, encodes them with ResNet-18, aggregates with a one-layer GRU, and predicts boss-player distance, relative angle, and facing. A skill branch is deterministic: from the action log it derives previous skill, time since each skill, and cooldown readiness. Engine internals are unused at inference.
Decision (L2) feeds that compact state plus natural-language skill mechanics to Gemma-4-E2B-it and asks for a short-horizon plan. The language model is not fine-tuned on the training set, so it cannot clone the dataset's action distribution. Only the first planned action is committed; later steps stay provisional (receding horizon). A deterministic cooldown check drops illegal skills. Skill duration is a lookup, not an LLM guess.
Control (L3) writes each entity as "[ACTION] while [MOVEMENT]" and repeats the phrase across the skill's fixed duration so every generation slot has a prompt. Direct Control maps keyboard and mouse to the player phrase. Director Control takes a high-level instruction such as "the player keeps evading while the boss stays aggressive," and the LLM plans both sides.
Generation (L4) fine-tunes Wan 2.2 TI2V-5B on the dataset for 70k steps at batch 16, then distills it with the three-stage Causal Forcing recipe into a few-step causal generator at about 20 FPS. New frames go back to L1.
The training set, BOSS-140K (Boss-fight Observations with States and Skills), has 144,631 clips and over 200 hours from 14 bosses in Game A (an anonymized 2.5D title with blurred frames), Hollow Knight, and The Binding of Isaac. A state-conditioned agent farms the data. Engine state is collection-only.
L1 reconstructs geometry on all three games. Relative error versus median engagement distance is 2.9% on Game A, 6.7% on Hollow Knight, and 8.4% on Isaac. Distance-bin accuracy is 0.807 / 0.772 / 0.904. Isaac's angle MAE hits 29.38 degrees, coarser than the other two. A task-trained encoder beats frozen DINOv2-S and VideoMAE-B on every geometry metric.
L2 changes its pick when skill descriptions change. Skill names are replaced with neutral IDs; only the mechanics text remains. Removing descriptions flips the first skill 52.0% / 90.0% / 55.4% of the time on Game A / Hollow Knight / Isaac (N = 200 / 200 / 121). Swapping descriptions flips 83.5% / 87.0% / 80.2%. The model follows the transferred mechanics 69.0 / 73.5 / 79.3 of the time and keeps the original ID only 16.5 / 13.0 / 19.8. Adding the rendered frame to the prompt does not help; mechanics-following drops 6.5 and 8.2 points on the last two games. The compact state card is enough. Extra pixels are not.
Closed-loop evaluation is Game A only: one-minute rollouts with 12 decision points, judged by GPT-5.5 and Gemini-3.1-pro, three passes each. All three systems share the Wan 2.2 backbone and the same training corpus. What differs is how NPC behavior is produced.
| Method | NPC behavior | GPT pref. / valid / sequence fit | Gemini pref. / valid / sequence fit |
| Wan without NPC control | implicit | 71.5% / 63.6% / 3.28 | 69.8% / 70.6% / 3.20 |
| Wan with NPC control | external | 70.9% / 63.6% / 3.22 | 70.3% / 70.9% / 3.17 |
| WorldMind | state-aware | n/a / 74.0% / 3.85 | n/a / 77.6% / 4.00 |
Preference is WorldMind's pairwise win rate against each baseline, about 70%. WorldMind's own step validity is 74.0% and 77.6%; sequence fit is 3.85 and 4.00 on a 0-5 scale. Baselines sit at 63.6% or about 71% validity, with sequence fit stuck near 3.2.
On the public WildWorld set, the Decision Layer shows partial cross-game transfer and stays state-sensitive. Compact-state reconstruction still needs target-domain adaptation.
Player control is already table stakes in game world models. NPCs are still "something that moves in the frame." WorldMind turns the NPC action into an explicit decision whose input is a small state card, not pixels. Anyone building a playable world model can copy the split: keep diffusion for pixels, hand planning to a small untuned LLM, glue them with a text action interface.
This does not replace a behavior tree. A 70% preference on one-minute, LLM-judged, single-title rollouts says explicit state-grounded decisions look more tactical than implicit video dynamics. It does not say the boss is ready to ship as a raid encounter.
There is no dedicated Limitations section. The gaps are visible anyway.
Closed-loop tests cover Game A only; Hollow Knight and Isaac stop at per-layer metrics. Judges are two LLMs. No human play, no human ratings. The geometry branch is trained once with seed 0, so there is no variance. Isaac's 29-degree angle error means the compact state is blurrier in a 2D bullet-hell view. Decision can partially transfer to a new game; reconstruction cannot, so geometry still needs labels.
L2 is sold as mechanics reasoning without fine-tuning, but the skill menu, boss identity, and mechanics text are all hand-written. What happens when the descriptions are sloppy is untested. Tactical quality is described as better than a random legal skill, with no number attached. Twelve steps in one minute cannot cover full-fight phase changes. Game A is fully blurred, so that slice of the data cannot be audited outside. The 20 FPS figure is the distilled generator; Decision still calls Gemma, and end-to-end latency is not broken out.