Spatial Memory Intelligence: Endowing World Models with Understanding-Driven Long-Term Memory
Ying Yang, Guiyu Zhang, Lianghua Huang, Chang Nie, Chenyang Si, Haofan Wang, Shaoshuai Shi, Li Jiang
cs.CV
2026-10-02
SMI fine-tunes Qwen3.5-4B to cluster, sparsify, retrieve, and filter video-world memory. On HY1.5 it drops 83.7% of memory and lifts scene consistency from 51.6 to 67.0.
Action-conditioned video world models write each new clip back into the history that predicts the next one. Three things go wrong as that history grows. Repeated views of the same place inflate storage. A revisit drifts in layout and object identity if the original geometry and appearance cannot be recalled. A flawed new clip, once stored, is reused as a reference.
Compression folds the context into a bounded representation and drops fine visual evidence with it. Geometry retrieval trusts camera or field-of-view overlap, so occlusion and pose error pull in frames that do not actually see the surface. Semantic retrieval mixes up places that look alike. Sparse attention saves compute and does not choose what to keep. Spatial Memory Intelligence (SMI) hands grouping, deletion, action-conditioned recall, and admission of new clips to an understanding model.
The decision model is a fine-tuned Qwen3.5-4B, which looks at frames and text and then chooses. The generators are WorldPlay's HY1.5-8B and Wan2.2-5B, with weights left unchanged. Camera pose only shortlists candidates.
Supervision comes from about a thousand real inference trajectories, each longer than a minute. People labeled 354 of them as demonstrations for GPT-5.6-sol and Gemini-3.1 Pro; Claude Fable 5 flags bad labels, and people correct them. The HY1.5 set has 63,911 examples, about 38.53% of them action-aware retrieval; Wan2.2 has 30,936. Fine-tuning is cross-entropy with AdamW, learning rate 2×10^-5, global batch 64.
Initial frames, actions, seeds, and context budget are shared. The field is Base with field-of-view overlap, compression via FramePack and Deep Forcing, sparse routing via MoC, geometry retrieval via VMem, and semantic retrieval via MemFlow. VBench and the GPT protocol each use 100 one-minute videos. Reconstruction uses another 100 trajectories that revisit a path, comparing outbound frames with the matching return frames. Three GPT-5.6-sol judges score blindly. Sparsity is the share of memory chunks removed; Base is 0%. Latency is end-to-end time relative to Base.
On HY1.5:
| Method | Scene consistency | Overall visual | PSNR | Sparsity | Latency |
| Base (FoV) | 51.61 | 53.15 | 12.48 | 0% | ×1.00 |
| VMem | 54.47 | 56.51 | 12.77 | 0% | ×1.47 |
| FramePack | 43.28 | 46.65 | 11.72 | 94.38% | ×1.00 |
| Deep Forcing | 37.68 | 33.63 | 11.26 | 79.21% | ×0.81 |
| SMI | 67.02 | 65.37 | 13.31 | 83.68% | ×1.38 |
Aesthetic quality moves from 0.445 to 0.469, and LPIPS from 0.603 to 0.555. Background consistency is 0.943, below Deep Forcing at 0.946. Against Base, scene consistency is 15.4 points higher and PSNR rises from 12.48 to 13.31. VMem, at 54.47, remains 12.5 points lower, with 0% sparsity and ×1.47 latency, slower than SMI at ×1.38. FramePack deletes more, 94.38%, and scene consistency falls to 43.28, below Base at 51.61. MoC scores 48.92 on scene consistency and 10.80 PSNR with no sparsification; MemFlow scores 43.32 and 10.30 PSNR at 49.43% sparsity.
On Wan2.2, SMI reaches 82.21% sparsity, 46.93 scene consistency, 51.49 overall visual quality, 11.56 PSNR, and ×1.31 latency. Deep Forcing's overall visual score is 50.73, 0.76 lower, at ×0.75 latency, with scene consistency 46.86. SMI background consistency is 0.917, under Base at 0.930. SMI does not lead every one of the nine GPT axes: long-range identity is 42.83 for SMI, 44.80 for Deep Forcing, and 45.92 for MoC; adjacent transition is 59.28 against Deep Forcing's 61.65. The clear gap is on HY1.5.
Ablations keep action-aware retrieval on. Removing clustering leaves sparsity at 80.53% but drops PSNR from 13.31 to 11.05. Removing reliability filtering leaves PSNR at 13.23 and cuts aesthetic quality from 0.469 to 0.456. With clustering, sparsification, and filtering all off, sparsity is 0% and PSNR is 12.81. Without clustering, sparsification deletes evidence from other places.
Qwen3.5-4B without this fine-tune reaches 86.34% sparsity, and PSNR falls to 9.40. Scene consistency is 42.84, below Base.
Twenty people picked a favorite on 15 scenes of 45 to 60 seconds. Best-choice shares for scene consistency are SMI 52.50%, VMem 27.00%, Base 11.00%, FramePack 7.50%. For generation stability: 57.83%, 27.83%, 11.33%, 2.00%. Deep Forcing and MoC were not included.
Explorable video has to keep an old place stable as the history grows. SMI leaves the generator weights alone and fixes the four memory decisions in a 4B model. On HY1.5 the latency is about ×1.38, 83.7% of memory is removed, and scene consistency moves from 51.61 to 67.02.
On Wan2.2, overall visual quality is only 0.76 above the faster Deep Forcing, and background consistency drops. The gain belongs to the memory controller, and its size follows the backbone. Understanding and generation are still trained apart.
The paper states three limits. SMI is not trained end to end with the world model, so the backbone's visual quality, control, and stability stay where they were. Separate representations add compute, especially early in a rollout while memory is still short. Reported latency is ×1.38 on HY1.5 and ×1.31 on Wan2.2. The task also does not cover long-horizon streaming understanding.
GPT-5.6-sol both labeled the training data and served as the main automatic judge, so the score lift is tied to that judge's taste. The user study has 20 people and 15 scenes and leaves out Deep Forcing. FramePack and Deep Forcing post the same sparsity on both backbones, 94.38% and 79.21%, which reads as a fixed compression ratio. The test videos last one minute, and round-trip PSNR stays near 13 dB. The runtime note discards scores below 6, while the training labels treat scores of 6 or lower as filtered. The shortlist of 7 geometric candidates and the cap on remote chunks have no sensitivity check.