AlayaVista Evolves the World in Panoramic Latents, Then Renders Only the Viewport

AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video

Jiaming Tan, Mingliang Zhai, Zhen Li, Yuwei Wu, Chuanhao Li, Kaipeng Zhang

cs.CV

2026-09-13

AlayaVista evolves a panoramic latent from one image and renders only the queried 1024×576 view, leading MoVerse and HY-World 2.0 on SSIM 0.4616 and rotation error 2.132.

What problem this solves

An interactive video world model has to keep the scene that just left the frame, follow a long camera path, and still emit a sharp observation with low latency. Those requirements pull the representation in opposite directions.

Perspective generators match the pixels a user sees, so anything off-screen has to be parked in long context, a cache, or a landmark bank. Full-sphere video generators keep angular coverage by synthesizing every direction at display quality, even though the user looks through one viewport. Explicit 3D pipelines such as Gaussian splats and meshes hold geometry, at the cost of lifting, maintaining, and rendering a scene asset.

AlayaVista splits the job. The world evolves as a panoramic latent. High-fidelity pixels are spent only on the requested perspective view.

Method

One perspective image, a camera trajectory, optional text. Four stages, all in latent space until a final RGB decode.

• Init: a frozen HY-World 2.0 panorama expander lifts the image to a 2:1 equirectangular (ERP) prior.
• Panoramic generator: Wan2.2-TI2V-5B on 48-channel WanVideoVAE latents, with SpheRoPE so the ERP seam is periodic and poles are less longitude-tangled, plus a parallel UCPE attention path that injects per-token ray frames. Streaming uses bidirectional attention inside a chunk and causal attention across chunks. A chunk is four latent slices, about 16 RGB frames at 16 FPS.
• Latent viewport renderer: maps 960×480 ERP latents to 512×288 perspective latents without decoding the sphere. The output is geometric bilinear sampling plus a global transformer residual and a local 4×4 correction.
• Perspective refiner: a deterministic upsampler builds a 1024×576 structural carrier, then a 4-step generative network restores detail. Both the panoramic generator and the refiner are distilled to four steps.

Training is staged, not joint. Data is MUGEN: 1,318 hours of ≥4K one-minute real panoramic clips from 6,446 YouTube sources, with trajectories, depth, instance masks, and hierarchical captions, plus Sekai2's panoramic subset. MUGEN-HQ is a 300-hour stratified slice.

Results

200 MUGEN-HQ cases, first frame plus target trajectory, 81 frames at 1024×576, against MoVerse and HY-World 2.0's final perspective RGB.

AlayaVista leads on SSIM, LPIPS, PSNR, Consistency, Quality, and rotation error. Dynamic drops to 0.9200 against 1.0000 and 0.9850; the paper attributes that to motion damped in rendering and refinement. Translation error 0.03312 beats HY-World 2.0 (0.03885) and loses to MoVerse (0.02746), whose Gaussian scaffold still tracks translation more tightly.

FVD is named in the protocol and missing from Table 1. The intro and conclusion claim long-horizon stability, revisit consistency, and streaming efficiency. The experiment section is this 81-frame table and two qualitative figures. No latency, FPS, or revisit number is reported.

Why it matters

For camera-controlled world models, off-screen memory becomes a property of the panoramic state instead of a separate retrieval stack, and the sphere is never decoded at display resolution. The backbone is Wan2.2-TI2V-5B; the initializer is an existing HY-World module. MUGEN is a sizable 4K real panoramic resource with geometry labels.

This is a systems compromise, not a new geometric primitive. PSNR 14.10 is still low in absolute terms. Errors can cascade across frozen stages.

Limitations

The authors note that a camera-centered panoramic state does not guarantee persistent spatial memory under large translations, nor strict geometry in highly dynamic scenes, and that panorama-expansion errors propagate.

The 200-case set is drawn from MUGEN-HQ, the same family as training. 81 frames do not support the long-horizon and revisit claims. Dynamic is worse; translation tracking loses to the 3D baseline. There is no end-to-end joint training and no isolated table for distilled rollout error. A code repo is linked; whether weights and MUGEN are fully released is not stated in the paper body.

Terms

• ERP: equirectangular projection, a 2:1 unwrapping of the sphere with a longitude seam and polar stretch.
• panoramic state: a sequence of panoramic video latents used as a camera-centered dynamic world, not as the display output.
• SpheRoPE: rotary positional encoding rewritten in longitude/latitude so the ERP seam is periodic.
• UCPE: camera poses injected into attention; here each ERP token carries its own ray frame.
• structural carrier: the upsampled low-frequency perspective latent that holds layout and motion while the refiner adds texture.

Source

• arXiv 2609.14462v1
• PDF
• Hugging Face
• GitHub · https://github.com/AlayaLab/AlayaVista

Related papers

• Code2Video-Compass: 2,104 Tasks Test 15 MLLMs on Writing Code That Renders Whole Videos
• Column shuffle self-pairs lift variable-length scene edits to 47.6% accuracy
• Same-time reallocation lifts HunyuanVideo sparse attention by 11.8 dB PSNR
• Frozen ALPS tokens hit 85.5% on LIBERO while matching V-JEPA 2-L
• Training-free segmentation: a frozen MLLM draws B-spline masks itself, 71.5 cIoU on ReasonSeg
• Wrong tap, same login screen: LGWM hits 0.987 AUC on GUI transitions in 17 ms

All paper explainers