AgentGarten: Code Worlds for Evolving Agents
Jiawei Chi, Shangchen Miao, Zhiyuan Shi, Kailu Wu, Hanyang Wang, Weiliang Chen, Qiyu Dai, Jinshan Ren, Jun Gao, Mingsheng Long, Yueqi Duan, Jiangran Lyu, Jialong Wu, Fangfu Liu
cs.CV
2026-10-09
Engines keep the rules; one renderer turns geometry into live views at 36.5 fps. Playbooks carry experience across rounds: shelters by round 4, bridge swaps from 71s to 41s.
What an agent can learn stops at the world it practices in. That world has to keep the consequences of actions, with rules that stay put, and the frames have to sit near the real visual distribution.
Simulators store collision, contact, and task variables in code, so the state can be inspected and edited. A new look still means new 3D assets. Video world models synthesize rich frames from data, but the state lives inside the generated history. Change one rule, or reshoot the same events under another appearance, and there is no separate record that says the episode itself was unchanged.
AgentGarten, from MirroS with Tsinghua University and Peking University, splits the job. An engine keeps persistent state and runs the rules. A shared neural renderer paints frames from the geometry that engine exports. The state update never reads pixels, so a bad frame changes the world only if the agent acts on it. One recorded trajectory can be rendered again from another reference image or camera.
A code world is a scene program, an engine, and a neural renderer. The engine applies actions to poses and task variables, then exports colorized depth or surface normals. The renderer adds an appearance reference, text, and visual history, and emits the current frame. The agent sees only those frames and acts with short Python programs. It never receives coordinates or the other player's view. A coding agent can write the scene from an image or from text, lifting the picture into coarse geometry and extending space past the input view. Probe cameras check contact and occlusion before the world is used.
The renderer starts from Cosmos 3-Nano. A frozen understanding tower encodes text; the generation tower paints frames. RGB and geometry share a frozen Wan autoencoder. Depth and normal tokens are concatenated with RGB tokens in one sequence and run through the same Transformer, with aligned space and shared temporal rotations. Joint attention is there so geometry and appearance can see each other without a channel-concat or ControlNet bias that every pixel must line up. Conditions are randomly dropped, downsampled, and noised.
Adversarial Forcing distills the model into a few-step, block-causal student. Each block conditions only on history the student wrote, and distribution matching pulls the rollout toward a frozen bidirectional teacher. Exact replay is what trains the history encoder. Pass one records each block's final denoising input and clean output with no gradient graph. Pass two recomputes history keys and values with the same block-by-block attention. Later losses update the parameters that wrote the cache, the sampler itself is not differentiated, and the recompute matches the rollout bitwise. Self Gradient Forcing recomputes the mask in one full-sequence attention call, which drifts in finite precision. A head on the frozen teacher backbone adds a relativistic R3GAN loss against real video, the piece that keeps long rollouts from filling with repeated texture. With the backbone frozen, the exact R1/R2 penalty on that head comes from one backward and one forward Jacobian-vector product, with no second backward through FlashAttention.
The cache keeps the reference and a window of recent frames. A distilled tiny decoder reconstructs pixels in under 10 ms, and frames leave over WebRTC. Weights stay fixed. Each round writes a playbook and hands it to a fresh conversation.
There is no FID or FVD. On one shared 30-second condition trajectory, Self Forcing grows repeated surfaces and loses detail; Adversarial Forcing still shows natural texture at 30 seconds.
Replay error was measured on one H100, 36 layers, 480 by 832, 61 latent frames, BF16.
| Replay | Relative L2 | Forward | Peak memory |
| SGF-style FlexAttention | 3.99% | 2.95 s | 50.83 GiB |
| Block-by-block SDPA | 0, bitwise | 2.79 s | 50.92 GiB |
Forward time drops 5.4%. Peak memory rises 0.2%.
Steady-state speed uses the same GPU and resolution, four denoising steps, and a cache of 5 sink frames plus 44 recent latent frames. One served block of 16 frames averages 438.8 ms, or 36.5 frames per second: 10.6 ms to encode conditions, 414.4 ms in the Transformer, 8.8 ms for the tiny decoder and host transfer.
Hide-and-seek starts from empty notes, five games a round, and the agents see only rendered first-person frames. Panel shelters appear by round 4. Seekers carry a ramp over a wall by round 10. The 2019 self-play RL agents, trained from random weights on object coordinates, built shelters after about 25 million episodes and used ramps after about 100 million. The paper treats those counts as a different learning regime, not a sample-efficiency ratio.
Four further worlds use the same loop for four rounds, one episode each.
| World | Measure | R1 | R2 | R3 | R4 |
| Companion dog | engagement | 13 | 14 | 19 | 19 |
| One-lane bridge | seconds to swap | 71 | 68 | 45 | 41 |
| Herding | score / sheep penned | 60 / 3 | 90.1 / 4 | 87.6 / 4 | 88.3 / 4 |
| Quarry loader | score out of 100 | 30 | 0 | 30 | 90.9 |
Both cars finish the bridge every round; time falls from 71 seconds to 41. Herding times out in round 1 with three of four sheep penned, then pens all four, while the score settles near 88. The loader scores 0 in round 2 and, in round 4, clears the rocks, delivers one, and parks in 329 of 360 seconds. Dog sessions last 60 seconds.
Depth or normals are a thin interface. Engines can export them, and real video can be turned into them, so several simulators share one renderer. A new world is code, and a log can be reshot in another style or from another camera. Bowling, a penalty shootout, and a crate puzzle already run this way in the browser: input advances the program, and the rendered stream is the only view.
There is no new reinforcement-learning algorithm. Notes pass to the next fresh conversation, the task file gives no solution, and the only observation is a rendered frame. Shelters and ramps in a handful of rounds are pretrained commonsense grounded in a closed loop. Evidence that the pictures are good is still mostly that 30-second comparison.
Geometry misses state that leaves depth and normals unchanged. A wheel can turn, and a symmetric body can spin, with the condition maps identical. Color and identity live in visual history, and a long occlusion or a revisit past the memory window can drop them. Downsampling also removes thin structure and small contacts.
Each extra world contributes one episode per round. There is no variance and no control that keeps playing without playbooks. Herding is not monotone, and the loader scores zero in the middle. The hide-and-seek rounds come from a single recorded run. Next to the 2019 episode counts they show a different regime; the paper already refuses an efficiency-ratio reading. Worlds are still written by hand. The frames are not scored against native engine renders or other real-time world models.