RecCAR closes reciprocal cross-attention gap in joint video diffusion models

barilan · hf · 2026-09-24

The study finds joint multimodal diffusion transformers are asymmetric in cross-modal correspondence: companion modalities (3D body motion, audio) strongly track video, but their reciprocal constraints on video remain weak — the "reciprocal correspondence gap."
RecCAR (Reciprocal Cross-modal Attention Regularization) is a KL regularizer aligning the weaker modality-to-video correspondence toward the stable video-to-modality distribution.
On joint video-motion and video-audio generation, Human Anatomy score rises from 0.69 to 0.75 and audio-video desynchronization drops from 0.804 to 0.752, with overall generation improved.

More from Multimodal