Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy
Abhinav Sharma, Sai Karthik Navuluru, Wang Wei, Daksh Dangi, Xiangbo Gao, Li Li, Bo Ni, Vardhan Dongre, Junda Wu, Xiyang Hu, Jiuxiang Gu, Seunghyun Yoon, Tong Yu, Chien Van Nguyen, Mohamed Elmoghany, Nedim Lipka, Hoda Eldardiry, Hongjie Chen, Tyler Derr, Thien Huu Nguyen, Zhengzhong Tu, Nesreen K. Ahmed, Franck Dernoncourt, Ryan A. Rossi
cs.CV, cs.MM, cs.SD, eess.AS
2026-09-28
Survey unifies joint video-audio generation, cross-modal generation, and coupled editing as three problems on one distribution; maps 29 methods on five design axes and 28 edit types.
Sound and picture are perceived as one thing: a door slam has to land on the frame where the door shuts, and a small offset reads as an error. Generative models for video and audio grew up separately, though, and two strong unimodal models combined naively produce streams that disagree. This survey, from 24 authors across Adobe Research, Dolby, UMass Amherst and ten other institutions, takes cross-modal coherence in time and semantics as its organizing question and covers three families of output: generating both modalities together, generating one from the other (soundtracking a silent video), and coupled editing, where a change in one stream propagates to the other.
Earlier surveys treated video generation, audio generation, or audio-visual understanding in isolation. None covers the pair as coupled, and joint audio-visual editing had never been systematically taxonomized. That is the gap.
The core device is a unified formulation: all three tasks are problems defined on a single distribution over audio-visual pairs. Joint generation learns p(v, a | c), cross-modal generation learns p(a | v) or p(v | a), and joint editing learns p(v', a' | v, a, e). The first two demand correspondence: the visible sources are the audible sources, and each acoustic event lands on the frames of its visual cause. Editing adds two constraints on top, propagation of the change across modalities and preservation of everything outside the target region. The voice-swap example makes it concrete: change a speaker to a child's voice, and the lip motion must follow the new audio while background and identity stay untouched.
On this sits a five-axis design taxonomy; every method occupies one cell per axis:
The axes are linked rather than free. A guidance-based strategy requires two frozen pretrained models, and every method that reuses pretrained backbones inherits continuous latents from them, so the discrete-token cells can only be filled by training from scratch or by coupling token-based unimodal generators, a route nobody has taken. The editing side contributes a taxonomy of nine categories and 28 edit types, from lip sync and joint content insertion to scene, narrative, and restoration edits, each annotated with its coupling direction (A+V, V→A, A→V). Development concentrates in joint content, synchronization, and text-driven control; the empty cells are the research agenda.
A survey runs no experiments; the numbers are its census of the literature. Across the 29 methods in Table 1 (25 joint generation plus 4 joint editing):
| Design axis | Dominant choice | Count |
| Audio representation | continuous latent | 28/29 |
| Video representation | 3D-VAE | 19/29 |
| Alignment mechanism | cross-attention | 19/29 |
| Pretraining | from scratch | 12/29 |
The empty cells carry as much information as the full ones. No joint method models either stream as discrete tokens, none generates audio directly in the waveform domain, and audio-to-video has only isolated dedicated attempts. Editing is sparser still: scene and environment edits, narrative editing (cuts, reordering, summarization), generative inpainting and outpainting, and joint restoration have no dedicated methods.
Evaluation is the weakest link. No shared benchmark exists for joint audio-visual editing; each paper evaluates on data it assembled itself, so results are mutually incomparable. The benchmarks that do exist expose two failure modes: AVGen-Bench finds strong audio-visual aesthetics paired with unreliable semantic grounding, and AV-Phys Bench shows joint models routinely fail physical-commonsense checks, with sounds mismatching the material or force of their visual events.
For anyone building audio-visual generation, this is a map with the blank cells marked. Token-based joint generation, audio-driven video, cross-shot narrative editing: the survey tells you no one occupies these cells and gives the structural reason each one is empty. For editing researchers, the absence of a shared evaluation set is itself a topic. The dataset table (from AudioSet's 2M clips to MMTrail's 20M-plus, 2M of them multimodally captioned) and the metrics table (CLIPScore, FAD, SyncNet's LSE-C/LSE-D, ImageBind scores) work as a ready-made selection list. No new model and no new SOTA; the value is organizational, since the unified formulation lets the three tasks borrow machinery from each other and the taxonomy makes the gaps visible.
The authors list four. The scope admits only methods with video or audio as an output and the other modality somewhere in the pipeline, excluding pure single-modality generation and non-generative audio-visual understanding. The strongest current systems, Veo 3 and Sora 2, cannot be placed on the axes because they disclose nothing, and appear only in an appendix on user-facing capabilities. Coverage runs to mid-2026 in a fast-moving field. And the five axes are complementary rather than mutually exclusive, so placement involves the authors' judgment; partially documented systems such as Wan 2.5 are marked only on disclosed axes. One more observation from this reading: the survey deliberately collects no performance numbers and offers no cross-method quantitative comparison, so it cannot answer which method wins. That is a trade-off, and also a ceiling.