Code World Model: Coding Agent as World Brain
Yiwen Chen, Guosheng Lin, Chi Zhang
cs.CV, cs.AI, cs.CL
2026-08-26
A coding agent keeps world state in code and compiles a proxy video to steer MiniMax-H3. Results after 5.6 hours of GTA fine-tuning are qualitative, with no numeric baselines.
Video world models treat an interactive world as "given an action, generate the next clip." Visual quality, long-horizon consistency, and memory have all improved. Video still records outcomes. In a game, each frame is the render of running fixed code, and that executable logic is thrown away after the pixels appear. The model has to infer rules from sparse visual consequences. Off-screen state has no continuous record. Causal chains that cross places and long world-time are thinner still. More video gives more outcomes, not the mechanisms.
An open world also needs consequences that persist. After a player assassinates a city's ruler, succession, order, factions, and NPC goals should keep evolving while the player is gone, and line up when they return. That needs a durable world state and reproducible rules, not another pretty clip.
Code World Model splits evolution from appearance. A coding agent is the world brain. It handles sparse, semantically heavy decisions: read state, infer consequences, pick or rewrite mechanisms, and emit executable code. Code runs the dense, repetitive, deterministic updates: positions, collisions, cooldowns, numeric rules. The agent does not step through the entire world. The world program can change; it is not frozen game logic.
Executable state cannot be dumped into a video model. Structured text does not steer camera paths and entity motion reliably. A full 3D asset pipeline is expensive and locks appearance too early. The compromise is a proxy: a handful of composable coarse primitives that record camera, entity pose, trajectory, and spatial relations, compiled deterministically into a low-resolution proxy video. In the implementation, spatial resolution is one quarter of the target, so visual tokens are about one sixteenth. Text still carries appearance, identity, and action semantics. The proxy carries the spatiotemporal constraints the current observation must obey. Fine motion and physical detail stay with the video model's priors.
Training pairs are recorded from gameplay in sync: target RGB plus the runtime fields needed to build the proxy. Full meshes, textures, and joint-level skeletons are dropped, because current coding agents cannot hold that layer. The prototype uses about 5.6 hours of GTA V across 157 takes, sliced into 9,420 five-second clips. On KITTI-360, calibrated poses and semantic reconstruction compile proxies offline, showing that real video can be aligned without action labels. The reported video-model fine-tune still uses game data.
At inference, GPT-5.6 Sol is the coding agent. It rewrites existing AAA scenes and gameplay templates rather than building an open world from scratch. The video backbone is MiniMax-H3 Ref2VA, with rank-128 LoRA on all 50 transformer blocks, about 596M trainable parameters, three epochs and 3,534 steps on eight H800s. Each call emits 124 frames at 1344×768, 24 FPS. GPT Image 2 supplies a first-frame appearance anchor from the first proxy frame and a text prompt. Long video slides 124-frame windows with a 34-frame overlap, adding 90 new frames each time.
Almost all evidence is qualitative. The model follows proxy-specified character positions, motion, layout, and camera paths while keeping detail and local dynamics. Training saw GTA; prompts still produce unrelated characters and styles. The proxy constrains spatiotemporal structure. Appearance comes from text and pretraining. Finer temporal comparisons sit on the project page, with a claim of tighter control than action- or camera-conditioned video world models. The paper itself reports no FVD, no control error, no user study. Those comparisons also ignore latency.
Real data only shows that a proxy can be compiled from reconstructed geometry. The KITTI pairs do not train the video model, and no real-domain generation metric is reported.
The split targets a structural hole in video world models: rules and consequences should not live only in pixel history. Code is inspectable, testable, and revisable. The video model still supplies appearance and motion priors. A proxy is closer to engine state than a longer text prompt, and cheaper than a full 3D sim. For interactive worlds or embodied training environments, this is a usable control surface: state in code, pixels from generation.
The evidence supports interface feasibility. It does not support open-ended evolution yet.
The authors list two. Compute is limited, so training is small, quality is limited, and there is no autoregressive real-time generator. Current coding agents still cannot implement complex game mechanisms reliably from scratch; the prototype never builds a complete open world on its own.
A few more gaps sit in the methods. There are no quantitative baselines. The coding agent edits existing AAA scenes, so a large share of world evolution is still the game engine. The first frame depends on a separate image model. Proxy resolution and primitive vocabulary are fixed; letting the agent switch the proxy on or off is described as part of a broader design space, not this paper. 5.6 hours of GTA with three LoRA epochs cannot carry the opening claim of approaching real-world causal complexity. The teaser figure is labeled as AI-generated illustration.