World-T3 lifts revisit PSNR from 13.52 to 18.31 dB at fixed memory cost

World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models

Junyi Wu, Zekun Li, Chenhongyi Yang, Chen Guo, Lingchen Yang, Yongan Zhang, Xinyi Zhang, Hongbing Hu, Bartek Rymkowski, Srinath Sridhar, Yan Yan, Xijie Huang

cs.CV, cs.AI, cs.MM

2026-10-13

World-T3 stores past views in fixed fast weights keyed by camera pose. Revisit PSNR rises from 13.52 to 18.31 dB, and a 4-step generator runs at 20.6 FPS on one B200.

What problem this solves

A video world model keeps generating the next view as the camera moves. A few seconds of that already look plausible. Longer interaction asks for something stricter: stairs, window frames, and lamp posts the camera has already passed still have to be there on the way back. Autoregressive video diffusion keeps that history in its own context window. Once the window slides on, the old view is sampled again from the prior. The clip stays locally smooth, and the place has changed.

Explicit 3D reprojection and retrieval over past frames both let storage and attention grow with the trajectory. A fixed bank of memory tokens stays bounded and dilutes the earliest observations as new ones are packed in. World-T3 writes what it has seen into a fixed set of parameters, so the read and the write do not scale with rollout length.

Method

The work comes from Meta Reality Labs, the University of Illinois Chicago, and Brown. The backbone is Wan2.2-TI2V-5B. Training uses 908K camera-annotated clips, 307K of them built for revisits. Static clips are replayed with the path folded back, synthetic cameras run closed loops, and objects that would change between passes are removed. Ordinary footage rarely retraces its route, so without those clips the memory never has to recall a place.

A bidirectional teacher learns camera control first. Geometry is a Plücker ray map: six numbers per pixel, direction plus moment, every pose relative to the first frame. The same viewpoint always produces the same map, and that map is later the memory key. Adaptive layer norm injects it into every DiT block. Only rank-256 LoRA on attention and the feed-forward layers is trained; the rest of the backbone stays frozen. Full-sequence attention gives a stable prior. The model is not causal, so it cannot roll forward interactively.

Stage 2 is block-causal. Each block holds three latent frames. Context is the first block, never evicted, plus the latest clean blocks, 21 latent frames in all. The sink block anchors appearance. Blocks that leave the window enter test-time training (TTT): a small weight update at inference that absorbs the new observation into parameters.

Each layer gets a two-layer MLP. Keys come only from the ray map. Values come from clean hidden states, so diffusion noise is not stored as scenery. W1 stays fixed and maps the pose key through a GELU. W2, the fast weight, is the readout updated online.

On a smooth path the pose keys are nearly collinear, and a gradient write keeps reinforcing the same direction. Recursive least squares instead solves a weighted ridge regression over every pose-content pair seen so far. An inverse correlation matrix whitens the new key, W2 takes only the residual, and the ridge term is 1e-3. Each step is the exact minimizer, so the stored state does not depend on write order. A query uses the target pose, and a per-channel gate adds the result to the hidden state. At deployment the backbone and W1 stay frozen; only W2 and the inverse-correlation state move, with no backward pass through the diffusion net. Once the window is full, the cost of a step stops growing with the video.

Stage 3 is there because Stage 2 still needs tens of denoising steps per block. Consistency distillation cuts sampling to four steps. Dual-teacher distribution matching then holds quality. The bidirectional teacher covers short-range appearance and motion. The memory-augmented teacher covers long-range consistency. Each teacher has a critic trained on student samples. Before the memory score is applied, both fast-weight states are filled from the student's own rollout, so the penalty is drift along the student's path. The first 399 iterations use only the bidirectional teacher. After that, both are on.

Results

The revisit set has 120 held-out clips: 160 frames, 10 seconds, 16 FPS, 1280×704, reversing at frame 80. Scored pairs sit 24 to 36 latent frames apart, past the 21-frame window. Every method gets the same first frame, text, and 6-DoF path, with no post-hoc alignment. Semantic similarity is the DINOv2 cosine between the outbound view and the return. Revisit PSNR is measured against the ground-truth frame.

MethodRevisit PSNRSemantic sim.Rot. errorFPSMemory
SANA-WM13.52 dB0.92003.344°14.248 GB
HY-World 1.510.84 dB0.856312.864°4.5110 GB
Matrix-Game 3.011.15 dB0.822511.179°7.176 GB
DreamX-AR11.66 dB0.87688.011°11.076 GB
LingBot-Fast10.59 dB0.85809.817°1.6144 GB
World-T3, 50 steps16.69 dB0.94761.308°4.454 GB
World-T3, 4 steps18.31 dB0.96101.075°20.654 GB

SANA-WM, four denoising steps plus three refinement steps, is the strongest external baseline on revisit. The others use three or four steps. Against it, 4-step World-T3 cuts DINOv2 distance from 0.0800 to 0.0390, gains 4.79 dB of PSNR, lowers absolute rotation error from 3.344° to 1.075°, and lowers camera-matrix error from 0.1200 to 0.0318. Paired tests give it the lower distance on 111/120 clips and the higher PSNR on 112/120. On 96/120 clips it beats every external baseline on both. Outbound-to-return SSIM moves from 0.4704 to 0.7201, and LPIPS from 0.3805 to 0.1278. HY-World 1.5 matches the student on the first visit, feature distance about 0.028, then jumps to 0.1867 on the return. The student's return distance is 0.0499.

Against the 50-step teacher, the student wins revisit PSNR on 91/120 clips, 16.69 dB to 18.31 dB, with no significant gap on the four camera metrics (p from 0.14 to 0.36). VBench falls from 75.15 to 74.52, and the adjacent-frame memory score from 0.9442 to 0.9258. Imaging quality is 71.46, against 76.29 for HY-World 1.5. That baseline leads VBench at 75.32 and posts only 10.84 dB of revisit PSNR. VAE-decoded ground truth itself scores 75.69 on VBench; the paper treats that number as stylized sharpness, separate from pixel distance to the reference. The student's first visit is 19.99 dB and the return is 18.31 dB, a drop of 1.68 dB.

Split into three yaw-sweep tiers of 40 clips, student semantic similarity stays at 0.9582, 0.9644, and 0.9603. Matrix-Game 3.0 falls from 0.8743 to 0.7737. On the widest tier, rotation error is 1.288° for the student, 2.241° for the teacher, 4.316° for SANA-WM, and 11.075° to 15.477° for the other baselines.

On one NVIDIA B200, with online memory updates, the 4-step model runs at 20.6 FPS and 54 GB peak. It is the only entry above 16 FPS. The 50-step teacher runs at 4.4 FPS, also at 54 GB. Ground-truth video still shows 0.194° of rotation error under the same pose estimator, which is the noise floor.

Why it matters

People turn the camera around. Redrawing a place once it leaves the window means there is nowhere to return to. Fast weights split memory from context length: the window is capped, and the parameter state does not grow with the session. A long walk does not require an ever-longer KV cache.

At 1280×704, 20.6 FPS on one GPU is the only result here that clears playback rate. Cutting 50 steps to 4 raises revisit PSNR rather than spending it. The loss is about 0.6 VBench points. HY-World 1.5 still leads that perceptual score and barely remembers the return. Test-time memory is prior work. The additions are the least-squares write, which stops a smooth path from stamping the same pose, and the second teacher, which keeps that memory inside a 4-step student. The 20.6 FPS figure is one B200 at 54 GB peak.

Limitations

There is no separate limitations section. The trade the paper does report is the drop in VBench and in adjacent-frame scores after distillation. Sign tests favor Matrix-Game 3.0 and HY-World 1.5 over the student on visual quality.

The quantitative revisit is 10 seconds and 120 clips from the authors' own held-out split, on a prescribed trajectory rather than free user control. The 30-second real-camera sequences are qualitative, and the return poses only approximately match. Revisit training replays static scenes and drops dynamic objects. Order-invariant least squares fits a static map. If an object moves, the later look should overwrite the earlier one, and invariance points the wrong way. A key made only of pose cannot separate those two states.

The main table also skips the explicit-3D, frame-retrieval, and token-memory systems named in related work. Constant storage is an architectural fact: W2 and the inverse-correlation state have fixed shape, and the context window is bounded. Whether residual error saturates after several minutes is not measured. Revisit PSNR mixes appearance error with missing the commanded camera, so 18.31 dB is not a pure texture score. Semantic similarity of 0.9610 remains short of the ground-truth self-similarity, 0.9982.

Terms

Source

What people are saying

Related papers

All paper explainers