ReWorld: An Interactive World Model with Long-Horizon Memory
Zhifei Chen, Luozhou Wang, Guibao Shen, Dongyu Yan, Shuai Yang, Tianshuo Xu, Yihua Du, Wei Wang, Tianyi Gui, Lianghua Huang, Yingcong Chen
cs.AI
2026-08-25
ReWorld trains control and memory on different windows, then fills a 12-chunk cache with pose landmarks. Rotation error 11.95°; a 64s out-and-back still regenerates the start view.
An interactive world model has to follow the camera, remember places it already showed, and keep streaming. The tension is structural. Control fits in a short window; memory wants unbounded history. Systems that train both in one attention stack see the usual trade: add action injection and control rises while revisit fidelity falls. ReWorld splits the windows in training and bounds memory at inference with a fixed KV budget.
The backbone is Wan2.2-TI2V-5B, streamed as causal latent chunks. Two conditioners: MRoPE folds camera pose into attention phase so similar viewpoints match across time, and a Plücker ray map adds the current command to patch embeddings. Of 24 heads, 18 see only the last 12 frames and 6 see the full causal past. Each step picks the global set from a pool of 12 random partitions, so neither skill binds to fixed heads. The inference cache is 12 chunks: one sink, five recent, six pose-retrieved landmarks. Chunks aging out of the window enter a landmark bank of at most 30, admitted by odometry; a full bank evicts the most pose-redundant member. Random chunk-drop during training makes sparse caches in-distribution. A four-step DMD LoRA compresses sampling and leaves the control path LoRA-free, so one backbone serves multi-step quality and realtime. Data: 220,724 pose-annotated clips from eight sources on one metric action scale, with a 20% palindrome window on UE footage for revisit supervision.
The control bench is 40 start images times 6 trajectories, 240 clips. Overall rotation error is 11.95°, ahead of LingBot-World at 12.59°, with the best camera-motion consistency. VBench mean is 0.850 versus 0.844 for Yume-1.5. At k=96 (about 16 s) revisit SSIM is 0.384 and DINO 0.932 with 615 px of outbound path; HY-WorldPlay scores higher but travels only 210 px, and a short path is easier to remember. Ablations: chunk-drop plus random head routing lifts revisit SSIM at k=384 (about 64 s) from 0.3387 to 0.3752; a sliding window under the same budget gets 0.3476, and full KV OOMs from k=288. Action injection versus MRoPE-only cuts RotErr 17.66° to 13.21° but drops revisit SSIM 0.3898 to 0.3376; routing keeps control at 12.94° and restores SSIM to 0.3752.
| Method | RotErr ↓ | VBench mean ↑ | k=96 revisit SSIM |
| LingBot-World | 12.59° | 0.841 | 0.251 |
| HY-WorldPlay | 14.66° | 0.842 | 0.427 |
| ReWorld | 11.95° | 0.850 | 0.384 |
Playable world models stall when following and recognition fight for attention. Splitting abilities by window at train time and retrieving landmarks by pose at deploy time needs no extra loss. A 12-chunk cache that still regenerates the start view at about one minute, where full KV already OOMs, is the shape that can ship. LoRA distillation gives the same 5B backbone a preview mode and a realtime mode. HY-WorldPlay's stronger short-path revisit scores are a reminder to read memory metrics with how far the camera actually moved.
The revisit protocol uses constructed palindromes, not free user roaming. HY-WorldPlay still leads SSIM/LPIPS on short paths; ReWorld's memory edge shows up once the needle has left the window. Control uses ViPE re-tracking plus Sim(3) alignment, so it scores trajectory shape, not metric meters. VBench is normalized to 32 frames and is not comparable to native-length numbers. Bank size 30 and cache 12 are untested on longer open worlds. The project was done during an Alibaba internship; the data mix leans on Unreal renders.