H3-World: Turning Language Understanding into World Control
Danze Chen, Zeqing Wang, Ziyue Lin, Xingyi Yang, Yeying Jin
cs.CV, cs.AI
2026-09-02
H3-World turns MiniMax-H3's coarse language control into temporally grounded character and camera commands with 8k gameplay clips, 10k LoRA steps, and 0.199% trainable parameters.
As video generators improve, language already steers characters and cameras in a coarse way. MiniMax-H3, for example, responds to "walk left, pan right" with roughly the right motion, zero-shot. An interactive world model needs more than roughly: this latent interval should execute this command, and the next interval should not leak the previous one.
Most systems add action embeddings, FiLM, or camera-geometry modules on top of the pretrained generator. That is a second control pathway, expensive, and it can damage generation quality. This Tencent, NUS, and PolyU paper keeps control in the text pathway the model already understands.
Each video latent covers a short RGB span. Keyboard state inside that span is aggregated into one latent-level action, split into a character clause and a camera clause, and written as a short sentence such as "the man walks backward and strafes left, camera pans right slowly." There are 9 character and 16 camera clauses, 144 pairs, 135 structurally valid. Training covers 83 of them and leaves 52 unseen. The top 20 pairs are 71.4% of prompts, the top 40 are 95.4%. That is a natural compositional test.
Each prompt is encoded on its own, then packed with static semantics, a first-frame condition, and video latents into H3's single-stream self-attention. Positional codes align the k-th instruction with the k-th video latent. Single-egress routing says an action span, as a key, can be read only by itself and by its matched video latent. Video latents keep H3's original bidirectional attention, so the effect can spread through motion after it enters. LoRA, rank 32, updates attention QKV, the output projection, and a two-layer token refiner. The backbone stays frozen. Training uses 7,872 clips of 124 frames from ABot-World-Explorer-500h, 10,000 steps, learning rate 1e-4.
The main quantitative probe is cumulative horizontal Farneback flow. With a fixed first frame and seed, a sharp left pan then a sharp right pan yields 0.0 then -17.3 under one global prompt (the leftward half is missing), near-zero flow with the per-latent interface and LoRA zeroed, and +52.7 then -106.0 for H3-World. Reversing the schedule gives -58.7 then +121.0. Under a constant direction, global prompting and H3-World separate almost equally (301.8 vs 300.5), so pretrained H3 already has coarse control; what it lacks is temporal binding.
Additive action vectors and FiLM on video features respond weakly and inconsistently to recorded controls on held-out clips. The text interface tracks character and camera changes together. With the first frame fixed, opposite strafes and fast versus slow pans produce distinct trajectories. An unseen forward-plus-pan-tilt pair executes both clauses on held-out gameplay and on out-of-distribution first frames.
A lot of the control interface may already live in a large video model's language pathway. The missing piece is temporal grounding, not a new action space. For interactive world models that means a few thousand aligned clips and 0.199% LoRA can tighten coarse language control into scheduled character and camera commands. Splitting actions into reusable clauses is also cheaper than encoding a full key chord as one vector, given the compositional gaps in the data.
Generation is a fixed 124-frame, 50-step clip. There is no persistent world state, real-time interaction, planning, or policy learning. Compositional and visual transfer rest on representative examples and an optical-flow diagnostic, not on success rates over the 52 unseen pairs, many scenes, and many seeds. The training distribution is badly skewed: 40 pairs cover 95.4% of prompts, so rare actions are unmeasured. Single-egress routing is a hard mask, not a learned router, and would have to be rebuilt on another backbone. All numbers are on ABot gameplay; whether the same interface holds on real-world video is untested.