HelixWorld: A Real-time Interactive Audio-Visual World Model
Lei Ke, Jiahao Pan, Zeyue Tian, Jiaming Wang, Haoyuan Huang, Kam Man Wu, Pengjun Fang, Hongyu Liu, Chenyang Qi, Lin Wang, Ruibin Yuan, Weijia Chen, Fangneng Zhan, Qifeng Chen, Wei Xue, Yike Guo
cs.CV
2026-09-30
HelixWorld streams 24 FPS video and camera-aligned stereo on one H800, scoring 41.8 on spatial consistency versus 12.6 for EchoWM.
Interactive world models already roll video forward from a keyboard or a camera path. Sound is usually added after the frames exist. Turn the head and footsteps, surface contact, and off-screen traffic should move with you. The stereo image should move too.
Cascaded video-to-audio models never see the 6-DoF camera trajectory or the user action, so the pan stays put, and they wait for a finished clip. Bidirectional audio-visual backbones denoise both streams together and expose no control. Convert them to causal rollout and earlier errors pile up in pixels and in sound at once. The training pool does not rescue this. World-model corpora are mostly silent, and in-the-wild audio is packed with music, voiceover, and dual-mono tracks that copy one channel onto the other.
HelixWorld starts from LTX-2.3. Video tokens and audio tokens share one Transformer and are trained with joint flow matching. A bidirectional teacher sees the whole clip. A few-step student sees only the past and is what runs interactively.
The raw pool is 4.1k hours: 1.8k real-world video, 1.4k game footage with engine-logged cameras, and 0.9k open-source clips. Shots under 5 seconds, fake stereo, subtitles, and watermarks are dropped first. Survivors are cut into 121-frame clips at 24 fps and 768×512, about 5 seconds, audio and video sliced in one pass. Qwen3-Omni removes talking heads, picture-in-picture, and score music, and keeps off-screen events such as thunder or a passing car. VGGT-Ω estimates real-world poses. Depth Anything V3 recovers metric scale. Game clips keep engine ground truth. Three captions are written apart: vision only, audio only, and a binding from visible emitters to their sound. What remains is 3.0k hours and 2.1 million clips, a 73.1% yield.
Continuous 6-DoF poses enter through PRoPE, which rewrites each frame's attention queries and keys with that frame's projection, so the network sees how the camera moved relative to the start. Discrete actions use 81 classes, nine bins of horizontal translation times nine bins of view rotation, added to the diffusion noise level and applied with AdaLN-Zero. Both signals hit the video branch first and reach audio only through cross-modal attention. Unfreezing is staged so pretrained visual features survive: control modules alone, then the video branch, then joint flow matching.
The student packs co-temporal video latents and audio tokens into blocks. Attention inside a block is bidirectional. Across blocks it is causal, with history in a sliding KV cache. Single-step predictions are first fit to the causal teacher's probability-flow endpoints. Self-forcing then trains on the student's own past. Distribution matching alone loses diversity and drifts on long rollouts. Online trajectory distillation integrates the frozen teacher from the student's noisy state to a clean endpoint and pulls the student velocity onto that flow. Each update takes this loss with probability 10% and distribution matching otherwise. A long-horizon pass extends rollouts to about a minute, with gradients only on the active window. Inference uses four denoising steps.
Visual scores are the WBench navigation split. Timing is one H800 at 768×512 and 24 fps, decoding included. RTF is steady-state time after the first block, divided by the remaining duration. Below 1 is faster than realtime.
| Method | Audio | RTF | Average | Interaction | Physical |
| Alaya-EVOKE-Turbo | no | 4.29 | 82.0 | 83.9 | 74.0 |
| EchoWM | yes | 1.98 | 81.0 | 87.9 | 70.1 |
| Zing-0.5 | no | 2.81 | 81.0 | 84.2 | 73.8 |
| HelixWorld | yes | 0.77 | 79.9 | 86.4 | 70.9 |
| LingBot-World v2 | no | 5.21 | 79.4 | 82.8 | 69.1 |
| SANA-WM | no | 0.79 | 76.0 | 82.2 | 61.9 |
| Matrix-Game 3 | no | 0.58 | 71.3 | 83.6 | 59.3 |
| LTX-2.3 base | yes | 14.61 | 74.2 | 66.4 | 64.9 |
HelixWorld averages 79.9. Alaya leads at 82.0. EchoWM and Zing sit at 81.0. LingBot v2 is 79.4. Scene following is 76.5, below the LTX-2.3 base at 85.2. Interaction is 86.4, behind EchoWM at 87.9 and well above the base model's 66.4. Among models that are actually realtime, Matrix-Game 3 averages 71.3 at RTF 0.58 and SANA-WM averages 76.0 at RTF 0.79. Both are silent. EchoWM is slightly stronger on the visual average and much slower: RTF 1.98, about 2.6 times 0.77.
HelixBench has 1,015 human-checked interactive clips. Spatial consistency runs from -100 to 100 and asks whether the louder channel is on the same side as the dominant on-screen source. The causal student scores 41.8, the teacher 33.2, EchoWM 12.6, and the LTX-2.3 base 1.0. Dubbing the same HelixWorld video scores -5.7 with AudioX, -9.5 with PrismAudio, and 14.8 with ThinkSound. FAD is 2.39, 3.08, and 7.75 for student, teacher, and EchoWM. KL is 1.39, 1.54, and 1.95. CLAP slightly favors AudioX, 0.309 against 0.302. DeSync is 0.59 s for the student, 0.40 s for the base model, 0.42 s for ThinkSound, and 0.67 s for EchoWM. Against its own teacher the student wins KL, FAD, ImageBind (0.299 vs 0.270), CLAP, and spatial score, and loses sync, 0.59 s against 0.50 s.
Progressive unfreezing cuts rotation error from 0.1237 to 0.1144 and translation error from 0.0957 to 0.0823. Three-track captions reach ImageBind 0.246, against 0.237 for vision-only text, a small lift. Trajectory distillation plus long-horizon tuning moves the ablation average from 77.9 to 79.1. Trajectory loss alone keeps clouds and landmarks more intact across 30-second rollouts, and raises DINOv3 diversity 21.7% and CLIP diversity 13.5% over distribution matching alone. In a blind test on 20 scenes, HelixWorld took a majority in every matchup among 212 audio votes. Among 190 muted votes it beat Alaya, HY-WorldPlay, and SANA, tied LingBot v2, and lost to EchoWM.
Visually this sits near Alaya and EchoWM and does not pass them. The spatial-audio gap is larger. A dubbing model can score below zero, which means the louder channel often sits on the wrong side of the frame.
Playable worlds, first-person previews, and simulators that need hearing require the sound field to move when the viewpoint moves. One H800, 768×512, decoding included, steady-state RTF 0.77, is fast enough to do that while the user is still steering. The trajectory loss ties student velocity to the teacher's probability flow. Other joint audio-visual models that need causal streaming can use the same kind of anchor.
The paper has no limitations section. The average still trails Alaya by 2.1. Physical plausibility is 70.9, against 74.0 for Alaya and 73.8 for Zing. Muted listeners prefer EchoWM. Audio is carrying a real share of the preference. Picture alone is second tier.
The spatial score is a left/center/right vote, with thresholds 0.2 on screen position and 0.1 on channel energy. Roughly the middle 40% of the frame counts as center. There is no source separation. Rain, multiple sources, and reverberation are outside the score. DeSync covers 165 event clips, and the student is worse than the teacher. Both the benchmark and the spatial score were defined for this paper.
Real-world poses come from VGGT-Ω plus depth estimation: 0.234° rotation error, worse translation than ViPE, and metric scale from the median of monocular depth. Only game footage has engine ground truth. Pose error enters both the action labels and the audio supervision. The 81-way action set only bins horizontal motion, pitch, and yaw. Trajectories with mean speed at 40 m/s, or acceleration above 20 m/s², are discarded, so fast motion never enters training.
Reported RTF excludes first-block latency and was measured on an H800. Training used six machines with eight H200s each, and LoRA rank 256 on the student. Consumer GPUs are not reported. Long-horizon tuning raises the average without lifting consistency and physical scores together. Past 30 seconds the evidence is mostly qualitative frames.