Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning
Hanyang Wang, Yimo Cai, Weiliang Chen, Jiawei Chi, Haowen Sun, Qiyu Dai, Yi-Hsin Hung, Xingzhuo Guo, Jinshan Ren, Runmao Yao, Ziwei Liu, Mingsheng Long, Yueqi Duan, Jun Gao, Jiangran Lyu, Fangfu Liu, Jialong Wu
cs.CV
2026-08-28
Code-as-World writes physics as executable code, recovers it with a propose-simulate-render-verify loop, then trains VLMs on verified worlds. The 9B model scores 55.4 mean MRA on QuantiPhy, above Gemini-3.1-Flash.
Vision-language models can narrate a falling cup and mention gravity. They rarely recover the mechanism: mass, contact, and dynamics written in a form that can be executed, edited, and measured. Pixel predictors can be right for the wrong reasons, because contradictory internal stories still produce a plausible next frame. 3D reconstruction keeps geometry without explaining why support removal causes a fall. Language is a poor container for friction coefficients and metric trajectories.
Code-as-World treats the missing object as an executable world: task-relevant physics written as code, then tested as a scientific hypothesis.
An executable world representation has three parts. Physical composition holds objects, sizes, mass, friction, gravity, and structures that participate in contact. Dynamic evolution holds initial states, forces, events, and duration. Visual appearance holds camera, materials, and lighting. Editing one part does not require rewriting the others. Execution aims for physical equivalence, not pixel cloning.
Text and video share a propose-instantiate-execute-render-verify loop. Text is parsed into entities, spatial relations, and intended outcomes, with physical priors filling gaps. Video is reduced to masks, depth, tracks, and 3D meshes. The agent writes a code hypothesis, compiles it into a simulator, rolls out a state trajectory, and renders images, depth, masks, and tracks. Mismatches trigger a local edit. The budget is five rounds; failure yields rejection rather than a pretty but unsupported world.
Verified worlds then supervise VLMs. Image-space training teaches pixel measurements of extent and motion. World-space training uses physical priors to calibrate monocular video into real size, displacement, velocity, and acceleration. A 27B reasoning variant writes the measurement-and-calibration trace before the scalar answer.
QuantiPhy-validation uses Mean Relative Accuracy (MRA) under the official protocol:
| Model | Size | 2S | 2D | 3S | 3D | Macro avg |
| Gemini-3.1 Flash | — | 49.4 | 47.5 | 61.4 | 61.1 | 54.9 |
| Qwen3-VL-32B-Instruct | 32B | 38.1 | 39.7 | 39.8 | 43.0 | 40.2 |
| Code-as-World-VL-4B | 4B | 45.4 | 55.4 | 45.8 | 56.0 | 50.6 |
| Code-as-World-VL-9B | 9B | 55.0 | 52.9 | 55.6 | 58.1 | 55.4 |
| Code-as-World-VL-27B (reasoning) | 27B | 48.7 | 62.4 | 60.5 | 62.8 | 58.6 |
The 9B direct-answer model already beats Gemini-3.1-Flash and every listed open-weight baseline. The 4B model reaches 50.6, well above same-size Qwen3.5-4B. The 27B result changes scale and protocol together, so the paper treats it as evidence the recipe scales, not as a controlled reasoning ablation.
On video-driven reconstruction, five loop rounds beat Best-of-5 independent samples at a matched evaluation budget on visual alignment, object IoU, trajectory ADE, and accuracy within 2% of the frame diagonal. Text-built worlds can be re-simulated after editing initial velocity or camera, then turned into more realistic video while keeping the coded motion.
Metric physical labels are scarce. Web video almost never comes with true speed or acceleration. Executable worlds are a scalable annotation path for monocular quantitative reasoning. For physical VLMs, editable simulation data, or counterfactual video, this is closer to mechanism than more descriptive QA.
A 9B model beating larger proprietary systems suggests the structure of the supervision may be worth more than extra parameters. The same code world can be edited and rerun, which neither pixels nor captions provide.
Simulators do not cover real physics. Small changes in terrain, contact, or materials can send rigid-body motion off course. If the true process sits outside the engine, the loop can accept a locally plausible program that is mechanistically wrong. QuantiPhy mainly tests monocular scale calibration for size, displacement, velocity, and acceleration. Contact, deformation, fluids, and long-horizon multi-body dynamics are out of scope.
The trained VLM consumes the products of discovery. It does not internalize proposing, simulating, diagnosing, and revising. The 27B gain is entangled with the reasoning protocol.