WM^MLLM: Turning MLLMs into World Models
Baiang Li, Enze Xie, Jincheng Yu, Jin Wang, Haozhe Liu, Yitong Li, Juanxi Tian, Felix Heide
cs.CV
2026-11-18
WM^MLLM stores the world as a panorama, finishes 85.3% of MBench-A round trips versus 50.9%, and raises local-edit completion from 34.8% to 83.0% via post-hoc region selection.
Video world models keep the scene inside the generator's context. Leave and come back, and the view is synthesized again, so an opened container or a new light can drift or vanish. In Figure 1, five video models finish a camera round trip on a different scene. The mean pixel gap between the first and last frame, on a 0-255 scale, is 88.4 to 101.3. WM^MLLM's gap is 1.1.
An interactive world needs three properties at once: a return shows what was saved, a local edit leaves every pixel outside its region alone, and the next request starts from the previous result. Meshes and Gaussian surfels can show new viewpoints and accept regional edits. Among systems with an explicit world state, none in Table 1 documents an MLLM planner, object edits, exact return, and a check before the edit is stored.
The state lives outside every generator. Each revision holds a 4096 by 2048 equirectangular panorama: base image B, composite panorama E, and native-resolution patches Q, along with a scene graph, object records, and facts taken from images. The revision number increments only when the physical content changes. Turning the camera does not increment it. A deterministic ray sampler reads the panorama and calls no model, so a revisit at the same revision, camera, and profile matches the stored view pixel for pixel.
GPT-6.1-sol turns text or a box into an ordered list of operations and sends them down one of five routes. Rotation and revisit only render. A local edit asks GPT-Image for a candidate crop and writes it inside a support region. Lighting, weather, and season replace B and E and retire older patches, which would otherwise paint the old appearance back in. Walking to a new place creates a new scene node. Sol-H3 plays the transition from the two endpoints and never writes the state. Each endpoint gets at most two image proposals and one video.
The support is chosen after the image exists. Generators move subjects past a box predicted beforehand, and shadows and contact spill outside it, so the early box clips the change. A selector compares the source with the proposal, marks the old footprint, the new footprint, and those effects, dilates the mask by 16 pixels, and feathers the edge so the blend reaches full weight 12 pixels in. If any panorama pixel outside that mask changes, the write aborts. There is no fallback to the planner's box.
The checker scores source description, action, requested change, and scene identity. A Fail is retried once with the failure record attached. A second failure keeps the previous state, and the rejected pixels never enter memory. An Uncertain score does not block the commit. The committed endpoint is the input of the next request, which is where return, locality, and persistence come from.
On 224 shared MBench-A items scored by one evaluator, WM^MLLM has the highest mean on all four environment-recovery metrics against six action-conditioned video world models. The 95% intervals do not overlap any baseline.
| Metric | WM^MLLM | Best baseline |
| Epipolar | 91.5 | 74.0 (HY-WorldPlay) |
| Reprojection | 92.2 | 75.6 |
| Lighting | 99.4 | 98.6 |
| Style | 99.3 | 97.9 |
| Round trips done | 85.3% (191/224) | 50.9% |
A completed round trip turns at least 15 degrees with the request in the first half and ends within 10 degrees of the start. The median turn is 43 degrees, and 189 of 224 items finish within 0.1 degree. Baselines complete 0.4% to 50.9%, with median turns of 82 to 173 degrees. Median LPIPS between pose-matched outbound and return frames is 0.037, against 0.32 to 0.65. HY-WorldPlay scores higher on camera motion, 92.5% versus 82.5%. A blind check of 40 videos counts a joint success when the camera goes back and the view is usable: 31 for WM^MLLM, at most 13 for any baseline. Infinite-World and Yume-1.5 come back to about 8.2 and 4.9 degrees and still show a different counter.
Of 454 segments, 88.3% are geometric, 215 rotations and 186 scene restores, and the generator is not called. Drop items with a hold or a terminal failure and Epipolar is still 93.9, against at most 74.5.
CompBench runs on 2,709 GPT-Image proposals. All 15 metric columns are the best of 15 baselines. Background preservation is the wide margin: PSNR is 30.440, 29.104, and 32.868 on local, multi-object, and implicit-reasoning edits, against at most 27.692, 26.849, and 28.891. Foreground LC-I sits just above BAGEL, at 0.845 versus 0.838, 0.911 versus 0.883, and 0.880 versus 0.874. On 150 proposals judged blind, keeping the whole proposal completes the most instructions, 91.3%. The share that is complete and free of extra changes is 52.6% for support compositing, 30.0% for pasting the whole image, and 22.0% for threshold paste. On multi-object LC-T, threshold paste scores 21.335, above 21.070.
The same registration and feathering on 112 local edits gives a blind completion rate of 34.8% when the support is predicted before generation, and 83.0% when it is selected after, a gap of 48.2 points with interval 37.9 to 58.3. All 46 revisits match their hashes. All 97 local edits that pass the check leave panorama pixels outside the support unchanged, and 92 of them are judged to match the requested region. Over 21 requests in one world, earlier edits remain, and the first 20 change 10.7% of the panorama. Strip the revision out of view lookup and 60 of 198 lookups return a view from before a later edit.
Walk, edit, and come back to the same pixels, starting from one image: those three demands are commit rules here. After a commit, a turn of the head or a revisit renders without a model call. The image model only proposes a view.
Picture quality still depends on GPT-Image and GPT-6.1-sol. Against video world models, the extra consistency is re-rendering state that is already stored. Against image editors, pixels outside the support stay put, and instruction completion is lower. The pace is minutes. Median state processing is 120 seconds for a rotation, 215 for navigation, and 267 for several operations. Request to first playback takes a median 609.4 seconds on two A6000 GPUs. A 124-frame transition takes a median 71.9 seconds on four H100s. In 20 paired sessions of eight turns, removing the observer's facts still keeps the edit on 32 of 40 returns and finishes 7 of 20 sessions, the same counts as the full system. The panorama is what carries the edit.
Requests often fail. Of 280 failure holds, 276 are rejected endpoints. A hold lifts appearance scores while the motion in the request never happens. The checker and the blind judge do not agree. The judge marks 190 of 309 accepted endpoints complete, and also 83 of 276 rejected ones. That judge is Claude Opus 5.5. There is no human calibration, and frames sampled from a clip do not establish the motion between them.
A bad endpoint is the next request's source. The following segment finishes in 100 of 153 cases after a correct commit and in 25 of 95 after an incorrect one. The panorama is the world around one camera center. Translation and unseen regions still require generation, and a seam is flagged in 208 of 227 environment items. A motorcycle asked to accelerate down an orange slide stays beside it, and the checker returns Pass. A coastal view asked to move right shifts the wrong way, the checker returns Uncertain, and that label still commits.
The 21-request sessions regenerate candidates that fail, so they are not a first-try success rate. A shorter planner prompt, and a variant that takes the last video frame as the next state, produce differences whose intervals include zero. Most segments here re-render a stored panorama. The MBench baselines have to synthesize video for the whole trip.