Marionette: Predicting World States, Rendering Geometry, Painting Appearance
Zian Meng, Zhen Li, Chuanhao Li, Qiang Li, Kaipeng Zhang
cs.CV, cs.AI
2026-08-15
Marionette splits an interactive game world model into a small net predicting an explicit 276-D state, a zero-parameter renderer computing exact geometry, and a video diffusion handling appearance only — no fidelity loss (FVD 831 vs 799 for recorded pose) and 66% fewer ground-penetration frames.
Interactive game world models autoregress visual observations directly in pixel or latent space. Pose, geometry, and occlusion stay implicit inside the same generative sequence, so errors accumulate over long rollouts and consistency with controllability break together: characters sink through terrain, the pair drifts apart, and there is nowhere to apply a fix because the world has no explicit representation.
Marionette cuts the work between prediction and exact computation: neural networks handle only what they are good at (dynamics and appearance), while every quantity that must be exact (metric geometry, root integration, joint kinematics, camera projection) goes to a zero-parameter deterministic render operator. Three stages:
Control gets clean as a result: to command an action, overwrite the action token at that frame, with no retraining and no auxiliary conditioning network. Long-horizon repair works the same way: two rules on the explicit state (a terrain collider and a separation cap) fix drift with zero changes to the observation model.
Data comes from engine-level recordings of Monster Hunter Wilds (the WildWorld corpus), each frame carrying synchronized RGB and 276-D state views. Dynamics trained on 1,395 segments at 20 fps (227.6 hours); observation on 4,008 clips at 704×1280 (673.8 hours).
| Metric | Marionette | Comparison |
| FVD16 (16-frame clips) | 831 | pixel-AR baseline 975; recorded pose 799 |
| Final-chunk FVD, long rollout | 1013 | baseline 1198 |
| Joint error after action swap (RA-MPJPE) | 0.357 m | correct actions 0.272 m, a 31% degradation |
| Two-character separation | 5.1 m with rule | 21.2 m without; 4.8 m recorded |
| Ground-penetration frame ratio | 0.114 with collider | 0.337 without, a 66% cut |
Two properties are verified separately. State layer: across 48 held-out segments, shuffling the action stream degrades joint error from 0.272 m to 0.357 m, worse on 33/48, so the predicted state genuinely consumes the action input. Observation layer: routing appearance through the predicted state scores FVD 831 against 799 for recorded pose, with overlapping bootstrap intervals, which the paper reads as no detectable fidelity cost.
This is a crisp demonstration of an architectural路线 for world models: instead of stuffing all structure into a latent and hoping the network learns conservation laws, model state explicitly, outsource exact computation to classical graphics, and let the neural parts do what they do best. For interactive content generation the payoff is threefold: controllability (overwrite a token to steer), repairability (add a rule on the state, never touch the generator), and interpretability (every one of the 276 dimensions has physical meaning). For embodied AI and simulation more broadly, the division of labor (constrain generation with a deterministic renderer) travels further than this paper's specific numbers.
The authors list three: appearance drifts over long horizons since geometry is exact but appearance relies on chunk-relay seeds; some entities in frame (AI companions, small monsters) lack recorded skeletons and become "unexplained" content for the observation model; and the observation model trains on recorded pose but consumes generated pose at inference, a distribution shift.
A close read adds concerns. The instantiation covers a single monster type and two characters, with action vocabularies of 173+689, far from a general game world model. FVD is a distributional metric with overlapping confidence intervals against the baseline, so "no fidelity loss" means "no loss detected," not "provably lossless." The penetration repair uses hand-written rules designed after knowing the failure mode; their generality is untested. And the zero-parameter renderer presupposes extractable skeleton and terrain representations, which most real-world video data cannot offer, and the method's precondition is itself a barrier.