OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation
Jun Zhan, Chen Yang, Yitian Gong, Donghua Yu, Kuangwei Chen, Wenbo Zhang, Kexin Huang, Qi Luo, Zhe Xu, Ying Zhu, Jin Wang, Tengyue Zhang, Qi Chen, Cheng Chang, Songlin Wang, Junqi Dai, Jiasheng Ye, Xiaogui Yang, Tianyi Liang, Xiangyu Peng, Zhaoye Fei, Shimin Li, Qinyuan Cheng, Xie Chen, Xinchi Chen, Xipeng Qiu
cs.SD, cs.CV
2026-07-27
OmniVAE jointly trains audio and video VAEs, aligning their latent spaces with segment-level contrast and per-modality distillation for better downstream audio-video sync and quality at no inference cost.
Generating synchronized video and audio from one text prompt is hard not because each modality is hard, but because they must line up: a dog's mouth moves when it barks, rain hits the right surfaces. Today's pipeline trains a video VAE and an audio VAE separately for reconstruction only, so the two latent spaces do not know each other and cross-modal sync must be learned from scratch by the downstream generator, which is hard and weak. OmniVAE bakes cross-modal alignment directly into the tokenizer.
OmniVAE keeps independent video (Wan2.2 backbone) and audio (DAC-style) VAEs whose reconstruction is untouched, then adds two training-only, inference-free objectives.
The first is segment-level audio-video contrastive learning. Each clip is split into 48 segments of 1/6 second. Video latents pass through a lightweight Transformer for spatial aggregation then temporal pooling; audio latents use ConvNeXt-1D along time; both land in a shared dimension. A bidirectional InfoNCE loss contrasts matched segments against negatives built at three levels (intra-clip, sibling-clip, cross-video, roughly 2:2:1), forcing fine temporal discrimination while preserving semantic diversity.
The second is per-modality semantic distillation. Frozen Qwen3-Omni vision (layer 27) and audio (layer 18) encoders act as teachers; a sigmoid-cosine loss distills their features into each latent space so each is easier to model alone. The two branches total about 1.08B parameters retained at inference; the contrastive head and projectors (49M) are discarded after training.
Training is staged: pretrain each branch on reconstruction only; jointly add contrastive and distillation objectives (memory-heavy, discriminators removed); then freeze the audio encoder and finetune only its decoder to restore adversarial supervision and fix mild electronic artifacts.
Downstream is verified with sync probing (predicting audio-video temporal offset) and real text-to-audio-video on Verse-Bench. Sync-probe A@1 on VGS-Sp rises from 6.4 (recon only) to 18.1 (+AVCLIP) to 20.2 (OmniVAE, frozen), reaching 22.3 after finetuning the aggregators. Contrastive learning is the main driver of temporal alignment.
Text-to-audio-video on Verse-Bench (CFG=5, mean of the last three checkpoints):
| Config | Desync lower better | LSE-C higher better | WER lower better |
| Recon | 0.884 | 1.479 | 0.243 |
| Recon+Distill | 0.814 | 1.450 | 0.205 |
| Recon+AVCLIP | 0.576 | 1.970 | 0.172 |
| OmniVAE | 0.570 | 2.093 | 0.168 |
AVCLIP cuts Desync and lifts LSE-C (lip sync) and audio-video semantic consistency; distillation improves audio quality (WER, IS). The two are complementary and strongest combined. Reconstruction is barely affected on video and only slightly traded off on audio, where OmniVAE is still the best tier.
Joint audio-video generation is the signature capability of closed systems like Sora 2 and Veo 3, and open efforts have struggled to keep up. This work locates the bottleneck at the tokenizer rather than the generator: instead of making the generator learn sync from scratch, the latent space already encodes it. For open teams building audio-video generation it is a plug-and-play foundation with released weights and code, at no inference cost.
There is tension between reconstruction and cross-modal alignment: adding contrastive learning noticeably degrades audio reconstruction metrics (STOI drops from 0.9981 to 0.9948), requiring a dedicated third stage to repair. All conclusions are verified on a single fixed downstream generator, and the authors concede the gains should be reconfirmed at larger generation scale. On Verse-Bench Set3, faces are too small at 256x256 generation to measure lip sync, so prompts were rewritten specially and are not directly comparable to the original. Validation is mainly on 8-second, 24 fps clips.