Precise Editing and Flexible Referencing for Interactable Worlds
Xinyao Liao, Xianfang Zeng, Zhu Liang, Zhoujie Fu, Qianxun Xu, Jiachi Liu, Gang Yu, Guosheng Lin
cs.CV, cs.AI
2026-09-28
EditWorld streams edit instructions and reference images during AR generation, scoring 73.8 overall and 80.0 editing on WBench-Editing vs 54.8 for YUME 1.5.
Video world models have mostly been navigation engines. Camera paths, keyboard and mouse, and text-triggered events all got better. YUME 1.5, LingBot-World 2.0, and DreamX-World can drop new events into a scene from text. XGEN-JING and ABot-World can lock identity from a reference at the start. What is missing is changing content that is already there: add a tower, strip a roof, turn asphalt into cobblestone, flip daylight into a blizzard, then keep moving while everything off-target stays put. References are stuck at initialization too. You cannot inject a car at second 15.
Event generation grows new content into empty space. Precise editing has to hit a named object, leave the rest alone, and stack later instructions on earlier results. Existing world-model benches mostly score navigation and visual quality. There was no shared exam for streaming edits.
EditWorld starts from LingBot-World-Base, a bidirectional video world model, in a collaboration between Nanyang Technological University and StepFun. An initial image is the world prior. The model then grows video chunk by chunk, conditioned on camera pose, text, and reference images.
Gated Causal Attention handles conditions that change over time. Every chunk always sees a scene description, the static content that should survive the edit. The edit instruction is visible only when that chunk is labeled during. To stop hard cuts when the instruction switches, the current chunk also attends to the prompts of the previous two chunks. Reference images are VAE-encoded and concatenated on the time axis, with one-way visibility: reference tokens attend only to themselves; a video chunk may attend to a reference only if its visible prompts mention that reference. Reference tokens also get a negative temporal RoPE offset, which pulls them off the video timeline and cuts down on copy-paste.
Sparse Context caps visual history. When generating the current chunk, the model always keeps the sink (the first chunk, which pins the opening world) and the two most recent chunks. Remaining history is ranked by cosine similarity of pooled pre-RoPE queries and keys, and only the top-k KV caches stay in context. Training always keeps sink and recent chunks and randomly samples the rest, so the model sees the same kind of sparse window it will get at inference.
Causal-only training drifts toward video continuation and dulls condition following. Autoregressive and bidirectional objectives are trained together, sharing parameters and data, differing only in the attention mask. Annealed self-resampling starts on clean ground-truth history and gradually swaps in the model's own rollouts. The realtime variant is distilled in two stages: Phased Consistency Model (PCM) brings each chunk down to 4 function evaluations (NFE), then Self Gradient Forcing distills the distribution under self-generated history.
The data pipeline is purpose-built. Navigation footage comes from Sekai, OmniWorld, and SpatialVID; editing pairs from Ditto-1M and OpenVE-3M. Global edits use a vocabulary of about 200 primary and 100 auxiliary attributes. Qwen3.6-27B picks attributes, Qwen-Image-Edit produces an anchor frame, and depth-controlled Wan2.2 fills the transition. Local edits start from VLM-filtered source-edited pairs and use the same transition fill. Each clip is annotated with a scene description, an edit instruction, before/during/after states, and cameras re-estimated by ViPE. For the reference subset, instructions are rewritten to strip content already visible in the image, so the model has to look at the reference.
WBench-Editing has about 150 cases, each 15-30 seconds (240-480 frames), with one to three streaming edit instructions. Some cases include references, and some later instructions depend on earlier ones. Editing is scored by VLM question answering on three questions: was the intended change made, does the world look right after the edit, and was unrelated content preserved. For a fair comparison, the main table uses text-only instructions and autoregressive variants of every model.
| Method | Overall | Editing | Navigation | Quality | Physical |
| EditWorld | 73.8 | 80.0 | 74.3 | 74.4 | 67.9 |
| YUME 1.5 | 67.0 | 54.8 | 72.9 | 76.1 | 62.0 |
| LingBot-World 2.0 | 66.1 | 47.2 | 70.7 | 74.4 | 66.2 |
| LingBot-World | 66.0 | 54.8 | 71.3 | 75.8 | 63.3 |
| DreamX-World | 65.9 | 53.9 | 70.4 | 73.3 | 63.0 |
The editing score sits 25.2 above the tied runners-up at 54.8, and overall is 6.8 above YUME 1.5. Physical at 67.9 is the best in the table. Navigation 74.3, Quality 74.4, Setting 60.9, and Consistency 83.8 are usable, not dominant: SolarWM leads navigation at 86.7, AlayaWorld leads consistency at 93.6, YUME 1.5 leads quality at 76.1.
On the reference-image subset, EditWorld scores 74.0 overall and 74.6 on editing, against XGEN-JING (bidirectional) at 68.7 / 63.6, SolarWM at 64.6 / 19.8, and ABot-World at 57.3 / 25.0. XGEN-JING can place a referenced cart, but the reference is locked at initialization, so the cart and a later cluster of yellow balloons show up together.
On the original WBench, bidirectional SFT averages 81.2, fourth overall, lifting LingBot-World (base-camera) from 78.5, with Setting moving from 72.6 to 84.6. Autoregressive SFT averages 77.7. The 4-step distilled model averages 79.4, with Setting 85.9 and Consistency 89.2, a 1.7 gain in average and 8.2 in Setting over autoregressive SFT. Editing did not come from sacrificing general world modeling.
Ablation: AR-only training gets 69.7 Editing Overall; AR+BI reaches 80.0. Detail Accuracy jumps from 34.9 to 53.2, the largest slice. Edit Cleanliness barely moves (78.3 vs 78.1). Unidirectional reference attention keeps fine details; bidirectional mixing dirties the reference representation. Dropping the two previous prompts makes instruction switches jump the entire scene.
A world model that can only be walked through is a generated game you cannot restage. EditWorld extends the control surface to add/remove/replace and style transfer, and lets a reference be injected mid-rollout. Teams building interactive worlds or game generation can reuse the gated conditions, bounded memory, and synthetic edit supervision. Code is public.
This is still an incremental stack. The backbone is LingBot-World, the labels come from off-the-shelf editors, and there is no new 3D world state. Worth following if you already run an autoregressive world model and need to change existing content. Too early to treat it as a drop-in world editor.
The paper has no Limitations section. The conclusion only claims a large editing gain with general world modeling still intact.
The exam is thin: about 150 clips, 30 seconds at most, scores from VLM questions, no human raters, no bake-off against general video editors. The main table also turns off reference inputs for fairness against text-only models, so it never measures the ceiling of streaming references. The reference comparison only includes three models that accept references at all. Sparse Context, self-resampling, and negative RoPE have no numeric ablations. The synthetic pipeline uses Qwen-Image-Edit and Wan2.2 as teachers; style leakage is unmeasured. Parameter count, train compute, and latency are unreported. Edit Cleanliness did not improve after adding the bidirectional objective, so clean edits that spare the rest of the scene remain weak. The reference subset's editing score of 74.6 trails the text-only 80.0.