Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds
Nan Duan, Haoyang Huang, Weiyang Jin, Haoran Li, Yaowei Li, Yuming Li, Yijun Liu, Xin Lu, Xiaoxiao Ma, Yanwen Ma, Yaofeng Su, Yilang Sun, Haoyu Wang, Zeyue Xue, Songchun Zhang, Junhao Zhuang
cs.CV
2026-08-24
JD's Echo-1.5 keeps character voice with composable cross-shot memory; its world-model variant scores 81.7 on WBench, ahead of HiDream's 80.7.
Video generators still behave like clip machines. Characters change faces across cuts, voices drift, and interactive worlds degrade once they condition on their own history. Navigation signals from keyboards, game logs, and web video also refuse to share a coordinate system. Joy Future Academy at JD built JoyAI-Echo-1.5 as one audio-visual backbone with two jobs: a long-video variant that remembers people and voices across shots, and a world-model variant that turns heterogeneous navigation into calibrated metric 6-DoF camera trajectories.
Capacity scaling does not fix forgetting, control mismatch, or rollout drift. The paper treats memory, geometry, and training-on-own-history as the actual levers.
Cross-shot memory is a composable condition, not a sidecar model. Each historical shot contributes one sampled frame as visual memory. Full-shot audio is speech-filtered before encoding, so background music does not get written in as a speaker. Target tokens, historical memory, and user-supplied memory sit in separated RoPE regions and do not share a timeline. Training randomly drops memory and the first-frame image, so one interface covers text-only, image-conditioned, memory-conditioned, and both. Stage I teaches retrieval from scene-disjoint clips of the same identity. Stage II adds high-resolution, subtitle-light, Chinese-heavy, transition-captioned data for quality and cuts.
The world model refuses controller-specific action spaces. Unreal Engine supplies ground-truth poses; internal games supply action logs; human gameplay and web video get metric 6-DoF tracks estimated with ViPE over roughly one-minute windows. Relative translations are rescaled by a dataset-level 90th-percentile displacement, then injected into visual attention through UCPE. Training is staged: audio-visual priors first, freeze the backbone and train only the camera pathway, then joint fine-tuning.
Both variants convert a bidirectional multi-step backbone into a causal few-step generator. Teacher forcing initializes chunk-wise causal attention on clean history. Short- and long-horizon Self-Gradient Forcing then trains on self-generated rollouts while letting later-chunk losses reach how earlier generated context is written into KV. Long-video distillation uses 8-step DMD with full video fine-tuning, audio LoRA, an RMS energy penalty, and an audio adversarial head so few-step sampling does not muffle speech or inflate loudness.
Long-video evaluation uses 100 stories and 3,000 shots. JoyAI-Echo-1.5 leads six of seven automatic metrics.
| Method | ViCLIP | Self-CIDS | Voice | Speech Recall |
| JoyAI-Echo-1.0 | 0.8026 | 0.7793 | 0.8129 | 0.9489 |
| LTX-2.5 | 0.7775 | 0.6640 | 0.7603 | 0.9545 |
| JoyAI-Echo-1.5 | 0.8264 | 0.7937 | 0.8524 | 0.9674 |
Against HappyOyster Directing, human raters prefer Echo on audio-visual sync (48.4% vs 20.3%) and story-instruction following (38.6% vs 17.7%). Identity is similar in 59.5% of pairs; when it differs, Echo wins 24.8% to 15.7%.
On the 158-case WBench Navigation split, the undistilled multi-step model scores 81.7 average, first; the distilled 4-step causal model scores 81.0, second; HiDream-O1-World sits at 80.7. On SANA-WM-Bench at 961 frames (60 s at 16 FPS), the undistilled model records 3.22° rotation error on Simple versus SANA-WM's 7.59°. On Hard, rotation error is 12.05°, worse than SANA-WM's 10.02°. Visual quality favors Echo on both splits.
For teams shipping narrative video or explorable worlds, the useful split is concrete: memory for identity, geometry for control, rollout training for drift. Code is public. The 8-step distilled long-video student and 4-step causal world model show that few-step sampling need not dump interaction score. This is a systems paper. It is not a drop-in block for an off-the-shelf DiT.
The authors flag accumulated rotational drift on hard long trajectories. On the Hard split the undistilled model’s 12.05° rotation error trails SANA-WM’s 10.02°. Against HappyOyster, Echo loses on semantic following and motion quality. Aesthetic score is only second among automatic long-video metrics. Speaker memory inherits speech-separation errors and can write them forward. Long-video numbers in Table 2 run with a Director Agent that plans shots and picks memory, so they are not a bare generator. World-model data is heavily game-engine; WBench Physical is 70.6, below HiDream’s 72.1.