All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation
Ohad Rahamim, Dvir Samuel, Idan Schwartz, Gal Chechik
cs.CV
2026-09-23
Video models attend strongly from video to companions, weakly the other way. RecCAR aligns reverse attention, raising anatomy 0.69 to 0.75 and cutting AV desync 0.804 to 0.752.
Joint generators now denoise video together with a companion stream such as 3D body motion or audio. The usual wiring is bidirectional cross-attention, so each stream can look at the other. The training loss does not care how much useful signal each direction actually carries. In pretrained models the video-to-companion map is already informative, while the companion-to-video map stays weak. The architecture is two-way. The information flow is not.
That gap undercuts the reason for joint generation. Motion can tell the video where a limb should be; audio can tell it when a hoof hits the ground. If those tokens barely update the video stream, the extra modality becomes a spectator. The paper writes both directions as distributions over the same video tokens and calls their mismatch the reciprocal correspondence gap.
RecCAR (Reciprocal Cross-modal Attention Regularization) is a KL term on existing cross-attention. No new modules at inference.
The two native attentions are not comparable. Video-to-modality already softmaxes over video tokens. Modality-to-video softmaxes over modality tokens. RecCAR re-normalizes the latter logits over video, so both maps answer: for this motion or audio token, which video tokens belong to it.
In a typical pretrained checkpoint the forward map localizes a head-motion token on the head; the reverse map often sits on the upper torso. RecCAR stop-gradients the stronger map and pulls the weaker one toward it. Total loss is the original generative objective plus this term with λ=0.01. The backbone stays frozen; only LoRA of rank 128 on attention projections is trained. The same recipe is applied to video-motion and video-audio, always treating video as the richer teacher.
Video-motion is grafted onto EchoMotion. Training pairs are self-generated then filtered: from about 18,000 candidates, keep those with MPJPE and NMPJPE both below 0.15, re-estimate 3D pose with CameraHMR, and end up with 4,292 pairs. Training takes roughly 48 GPU-hours on 4 H100s. The test set is 1,000 held-out VidProM prompts.
| Method | Human Anatomy | Dynamic Degree | Aesthetic |
| CoMoVi | 0.67 | 0.43 | 0.46 |
| FlowMo | 0.65 | 0.62 | 0.52 |
| EchoMotion | 0.69 | 0.83 | 0.56 |
| RecCAR | 0.75 | 0.83 | 0.57 |
Dynamic degree is unchanged, so the anatomy gain is not from slower motion. A 50-prompt ablation is sharper: ordinary fine-tuning on the same data drops anatomy from 0.70 to 0.66 and raises N-MPJPE from 0.197 to 0.208. RecCAR reaches 0.78 and 0.148. In the appendix, motion-to-video attention mass goes from 0.0113 to 0.0354, a 213% relative increase.
Video-audio uses LTX-2 and JavisDiT++, trained on about 4,300 VGGSound clips. On T2AV-Compass (500 prompts), LTX-2 absolute AV desync falls from 0.804 to 0.752, beating ITS, which is built for sync. On AVGen-Bench, LTX-2 goes 0.424 to 0.406 and JavisDiT++ 0.518 to 0.438. Audio quality, video quality, and semantic alignment stay flat or tick up. Standard fine-tuning barely moves DeSync.
A lot of joint-generation work adds bidirectional wires and stops there. RecCAR is a cheap check that both wires actually work: the teacher already lives in the other attention direction, no extra labels, no extra inference cost. If you already fine-tune LTX-2 or EchoMotion with LoRA, this is a drop-in regularizer. It is an incremental fix, not a new generator.
There is no dedicated limitations section.
The method assumes the video-to-modality map is a trustworthy teacher. If that map is wrong, KL copies the error into the reverse path. The paper never trains the opposite direction, and never tries companions such as depth or optical flow.
Video-motion data is generated by the same family of models, then filtered with an MPJPE threshold calibrated on 25 clips plus Gemini anatomy scores. That loop can hide distribution shift. The 0.75 anatomy number is on 1,000 prompts; the 0.78 ablation is on 50. Do not mix them.
On T2AV-Compass, JavisDiT++ absolute desync only moves from 1.078 to 1.046, while ITS sits at 1.040. RecCAR is not the winner on every backbone. Matching two attention maps is also not the same as matching semantics.