ForgeWM: Progressive Causal Training for Few-Step Action-Conditioned Video World Models
Xinye Li, Lingshuai Lin, Lei Wang, Liuzhou Zhang, Jialin Cui, Qingshan Li, Guanchu Wang, Qingbin Liu, Xi Chen, Jiang Bian, Wai Lam
cs.CV, cs.AI
2026-08-14
ForgeWM turns a bidirectional generator into 1/2/4-step causal world models. On Minecraft it leads imaging quality and key/mouse accuracy; the 1-step student runs at 72 FPS.
Action-conditioned video world models have to close a control loop: discrete keys and continuous mouse arrive at frame rate, and denoising must be cheap enough to keep up. Causal distillation can compress video synthesis to one or two steps, but game controls still have to stay aligned with temporally compressed latent chunks. At inference the model conditions on its own history, so visual error, action caches, and key–value caches feed the next chunk together. Fidelity, controllability, and multi-chunk stability are coupled.
Existing systems either fold controls into camera poses or teacher-force on clean history, then drift once rollout is autoregressive. ForgeWM converts a bidirectional action-conditioned generator into few-step causal students without changing the keyboard–mouse interface.
Initialization is Matrix-Game 2.0. The VAE packs four video frames into one latent frame; a three-latent chunk spans twelve frames. Discrete actions enter as cross-attention keys and values; windowed continuous actions fuse with visual features before temporal attention. During rollout, visual, keyboard, and mouse caches update together.
Four stages change temporal execution, objective, and history distribution.
Deployment splits into two paths. Online interaction uses the low-step student. After the session, saved drafts are re-noised at ri=0.3 and refined with four updates of the same checkpoint, polishing the experienced trajectory instead of sampling a new one from noise.
Minecraft evaluation uses 77-frame rollouts with shared first frames and controls. Reference-aligned metrics use 1,000 paired trajectories.
| Model | IQ↑ | LPIPS↓ | KCtrl↑ | Mouse↑ | FPS↑ |
| ForgeWM-1 | 0.6776 | 0.6529 | 0.9545 | 0.7848 | 72.10 |
| ForgeWM-2 | 0.6865 | 0.6171 | 0.9740 | 0.8268 | 50.31 |
| ForgeWM-4 | 0.6788 | 0.6168 | 0.9740 | 0.8102 | 32.47 |
| Matrix-Game 2.0 | 0.6282 | 0.6443 | 0.9156 | 0.7061 | 32.35 |
| HY-WorldPlay | 0.6133 | 0.6172 | 0.9286 | 0.5818 | 7.54 |
ForgeWM owns six of seven quality/control columns. Quality is not monotone in step count; one-step throughput is highest. In a 41-person, 615-trial three-way study, ForgeWM-4 takes 68.8% of visual, 57.6% of action, and 55.6% of spatiotemporal votes, 60.7% pooled. Freezing ForgeWM-1 and adding solver steps leaves directional control stable; extra steps mostly buy motion magnitude and latency. Replay refinement scores LPIPS 0.6155 against 0.6168 for native four-step generation, with draft distance 0.1970 versus 0.6187 for from-noise regeneration. The same recipe transfers to gamepad FPS: macro LPIPS 0.656, motion magnitude 1.45× the reference.
The product shape is play at low latency, polish later. ForgeWM turns 1/2/4 steps into three operating points and reuses the online checkpoint for replay, so a second refiner is optional. Keyboard and mouse stay frame-aligned rather than being reduced to camera pose, which fits Minecraft-like controls. Only two interactive baselines are compared, and the controlled numbers are Minecraft-only. The recipe transfers; the leaderboard does not automatically.
Outside the 77-frame window, rollouts at about 3.5× that horizon slowly lose block structure and pick up spreading color artifacts. Indefinite stability is not claimed. On the FPS split, motion overshoots: magnitude lags directional fidelity. At the four-step budget, Stage 3 raises imaging quality without improving paired LPIPS versus Stage 2, so the extra stage is metric-dependent. HY-WorldPlay uses a different control parameterization through a deterministic adapter, so motion scale is not directly aligned. Broader out-of-distribution generalization is out of scope.