EchoWM: Open and Enterable Omnimodal World Models
Songchun Zhang, Yaowei Li, Junhao Zhuang, Weiyang Jin, Haoyu Wang, Xin Lu, Yilang Sun, Shiyi Zhang, Haoran Li, Xiaoxiao Ma, Yuming Li, Yijun Liu, Yaofeng Su, Yanwen Ma, Haoyu Wu, Zihan Su, Yue Ma, Lvmin Zhang, Haoyang Huang, Zeyue Xue, Anyi Rao, Nan Duan
cs.CV
2026-08-24
EchoWM maps keys and poses to one relative 6-DoF trajectory and jointly generates 720p video plus sound, scoring 81.7 on WBench Navigation vs HiDream-O1-World's 80.9.
Most world models still emit silent video, and the control surface is glued to a specific gamepad or camera rig. Game models eat WASD. Camera-controllable video models eat Plücker rays. The two condition spaces barely overlap. Audio is often painted on after the frames exist, so a user who walks into the scene never hears footsteps, collisions, or speech.
EchoWM targets an enterable omnimodal world: given a reference observation and a structured scene description, the user keeps navigating while the model jointly emits 720p video, environmental sound, music, and speech, with multi-turn continuation. The work comes from JD's Joy Future Academy with HKUST, PKU, HKU, Tsinghua, and Stanford.
The interface is camera intent. In first person the trajectory is observer ego-motion. In third person the same trajectory is a request for how the view should evolve; character locomotion and camera-follow are learned from data. The network never sees a character path, a controller state, or camera-rig parameters. It sees a relative 6-DoF pose.
Discrete keys and continuous metric poses are both converted into an SE(3) displacement relative to the first frame. Translation scale is a single dataset-level constant: the 90th percentile of per-clip maximum translation. Per-clip normalization would erase amplitude and can jump speed at chunk boundaries. Geometry is injected with Unified Camera Positional Encoding: relative ray transforms enter a parallel branch of video self-attention, zero-initialized so the pretrained pathway is intact at the start of control fine-tuning. Audio tokens get no direct trajectory condition.
Four data sources supply complementary holes. Internal scripted gameplay has action logs and a native stereo mix after HUD removal. Human-played internet gameplay has messier timing and richer speech. Unreal Engine rollouts have ground-truth metric poses and no natural audio. General internet video widens appearance and acoustics; only pose-filtered continuous clips enter trajectory training. The processed corpus is split into three mixtures: AV-rich for the audiovisual prior, control-clean for identifiable motion (28,605 trajectories in the coverage plot), and their high-quality intersection for joint fine-tuning. Narrative captions are stripped during trajectory stages so motion cannot leak through text. Issued keypresses and realized camera paths are stored separately: a forward command against a wall may produce almost no translation. The model is conditioned on the realized path.
Training has four stages. AV-CPT continues pretraining the full backbone without trajectory. Action-SFT freezes that backbone and trains only the camera branch. Joint-FT updates both at a lower learning rate. Autoregressive post-training turns bidirectional diffusion into causal chunks with teacher forcing, then short-horizon and long-horizon Self-Gradient Forcing, with Distribution Matching Distillation down to a four-step sampler. Long rollouts keep a sink-plus-FIFO KV cache rather than an external retrieval memory. The distilled causal model is EchoWM-Flash.
The interface covers navigation and viewpoint. Jumping, attacking, and object manipulation are out of scope.
Public numbers are close at the top, then spread on long-horizon pose.
| Setting | EchoWM | Closest strong baseline |
| WBench Navigation average (158 cases) | 81.7 | HiDream-O1-World 80.9 |
| Interaction on the same split | 87.2; Flash 87.9 | HY-World 1.5 AR-distill 86.8 |
| Consistency | 89.8 | LingBot-World base-camera 89.9 |
| SANA-WM-Bench short Simple VBench | 83.91 | SANA-WM 81.75 |
| Long Simple rotation error | 3.22° | SANA-WM 7.59° |
| Long Simple revisit PSNR | 15.10 dB | SANA-WM 14.16 dB |
Genie 3 scores 73.9 on the same Navigation split. Flash lands at 81.0 overall: 0.7 down from the multi-step model, 0.7 up on Interaction. On short Hard trajectories SANA-WM has lower rotation error (1.248° vs 1.697°); EchoWM keeps the visual lead (83.96 vs 81.79). On long Hard, rotation error grows to 12.05°. Accumulated drift is still the main failure. Under the 961-frame causal protocol, EchoWM-Flash reaches Simple VBench 80.13 and rotation 5.009°, against Evoke at 80.11 and 9.859°.
A 200-case internal user study mixes games, humanoid robots, outdoor, indoor, and driving scenes (132 third-person, 68 first-person). Overall preference is 46.50% vs 21.57% against LingBot-World-v2 and 63.13% vs 27.06% against HappyOyster. Motion is often judged bad on both sides. Against HappyOyster, EchoWM wins only 13.63% of semantic-following votes versus 29.44%. The training mix has no 2D/2.5D games and no robot data; qualitative rollouts still track viewpoint in those domains, without a quantitative table.
For a playable generative world, the useful move is to lift interaction off controller semantics and onto observation intent, so first- and third-person share one pathway. Audio is native, not a post process. Flash shows four-step causal distillation keeps most of the navigation score. The usable scope stops at navigation and viewpoint. This is not a game engine.
The public average lead is a fraction of a point. The harder evidence is long-horizon translation error, revisit consistency, and trajectory following after causal distillation.
The authors are direct. The interface does not represent jumping, attacking, manipulation, or robot commands, and it does not implement collisions or deterministic engine transitions. There is no explicit persistent 3D memory, so geometry, identity, world state, and audio can drift across continuations. Trajectories larger than the global translation filter are dropped; bigger translations, higher speeds, and unusual rotations are untested. Pose estimates from internet and gameplay video remain noisy even after ViPE, VGGT-Omega, and MoGe-2, and can correlate with content and motion blur.
Audio has no separate action-to-sound objective. Motion quality and some semantic-following votes do not favor EchoWM. WBench rescales trajectories for open-domain scoring, so absolute displacement has to be read from the speed-response setup; the main text only shows qualitative approach-to-subject comparisons.