FATE: Frame-Level Audio-Visual Temporal Embedding
Kaisi Guan, Bingzi Zhang, Xihua Wang, Ying Ba, Xin Cheng, Yijing Chen, Ruihua Song
cs.MM
2026-08-02
Existing audio-visual models either capture semantics (what sounds) and lose time, or capture synchronisation (when) without a reusable semantic representation. FATE keeps frame-level sequences from both modalities, aligns them on the physical timeline, and scores similarity over aligned frame pairs, trained with cross-video semantic and within-video temporal contrastive learning. It leads temporal retrieval by a wide margin, matches supervised methods on zero-shot event localization, and correlates best with human judgement as a generation-evaluation metric.
When a dog opens its mouth and barks, a person naturally knows what the sound is and when it occurs. Giving an audio-visual model that ability requires a representation that captures both semantic and temporal alignment at once. Existing methods lean to one side: embedding models pool each modality into a single vector for semantic matching and discard time, while synchronization models predict a temporal offset but carry no reusable semantic representation. FATE aims to bridge that gap.
The core idea is not to pool. FATE keeps the frame-level token sequences of both modalities, aligns them on the physical timeline with nearest-neighbor interpolation (audio as the temporal anchor), and defines similarity as the mean inner product over strictly aligned frame pairs. It compares both "what" and "when."
The training objective combines two terms: cross-video semantic contrastive learning (pulling same-modality content from different videos together and pushing dissimilar pairs apart) and within-video temporal soft-contrastive learning, which uses a raised-cosine kernel to assign intermediate-strength labels to nearby offsets rather than a binary verdict. This avoids the conflicting gradients that hard negatives produce.
FATE leads across three tasks.
Temporal cross-modal retrieval (find the video for an audio clip, or vice versa), on AVSync-15:
| Method | Intra-video V2A R@3 | VGG-Sync inter-video V2A |
| PE-AV | 18.00 | — |
| Synchformer | 38.89 | 10.78 |
| FATE | 55.74 | 37.76 |
Zero-shot event localization (AVE dataset, average accuracy): FATE reaches 48.3%, above the fully supervised DAM (47.8%) and far above the prior zero-shot PE-AV (31.8%).
As a generation-evaluation metric (correlation with human MOS): FATE scores 17.24 sample-level and 44.41 model-rank correlation, beating the runners-up CAVP (14.15) and AV-Align (24.99) respectively.
Ablation confirms frame-level design is the crux: removing it drops AVSync-15 intra-video V2A from 55.74 to 29.11 and inter-video from 34.13 to 19.33.
For anyone building audio-visual generation (video dubbing, text-to-audio-video), FATE offers a single embedding that judges both "is the content right" and "is the timing right," usable directly as a generation-quality metric instead of running a semantic model and a sync model separately. For audio-visual understanding (retrieval, event localization), it shows that keeping the timeline plus a dual-objective contrastive loss is a stronger route than global pooling, and matching supervised methods zero-shot suggests supervised data may be overrated on these tasks.
The authors concede two points: as a generation-evaluation metric, all metrics including FATE show only modest absolute sample-level correlation, so fine-grained synchronization evaluation of generated content remains an open problem; and the model-rank correlation is computed over only five generation models, so it is indicative rather than conclusive. One more open question: the paper centres "frame-level," but the compute and memory overhead of frame-level representation relative to pooled models is not quantified here, so the real deployment cost is unclear.