UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos
Yuxuan Zhang, Haozhong Xiong, Jiayi Song, Jinpeng Yu, Yang Shi, Jiaming Liu, Ruihua Huang, Liwei Wang
cs.CV, cs.SD
2026-08-12
UniSwap swaps a talking-video character's face and voice in one audio-visual diffusion model, streaming at 13.6 FPS on one H100, about 10x the fastest cascade.
Talking-video character replacement swaps the person speaking in a video for someone else, given a reference image and a reference voice, while preserving the original motion, scene, spoken content, and the alignment between lip motion and audio.
The standard approach chains two independent models, one for the face and one for the voice. Because they are optimized separately, no joint objective forces the generated lip motion to match the converted voice. Most diffusion-based methods also require the entire clip before they can generate, so they cannot stream block by block.
UniSwap is built on the frozen LTX-2.3 audio-visual diffusion model, with video and audio as two modality streams on a shared physical-time axis, coupled by cross-modal attention. It is the first framework to swap appearance and voice jointly in a single model.
Aligned cross-identity training pairs do not exist, so the authors build a swap-and-reconstruct pipeline. From an ordinary talking video they estimate 2D poses, remove the person to leave the background, and composite the rendered pose sequence back as the source, preserving motion and timing while suppressing appearance. On the audio side, Seed-VC converts the timbre toward a random speaker, preserving content and prosody. The reference is a portrait frame plus a random 30% audio crop; the original clip is the reconstruction target.
Three training stages:
At inference, Feature-RoPE Decomposition keeps cached positions within the training range: an Adaptive Sink Block permanently retains the first block as an identity anchor, reference keys are re-rotated at the current slot, and rolling blocks map into a fixed 4-slot window.
Short-video benchmark (10-second clips, 100 clips). Audio-visual sync: UniSwap reaches Sync-C 3.633, versus 3.289 for the best cascade (SCAIL-2 + Seed-VC), so the generated lips line up better with the audio.
| Metric | UniSwap | Best baseline |
| Sync-C higher | 3.633 | 3.289 (SCAIL-2+Seed-VC) |
| Identity DINO-S higher | 0.629 | 0.630 (SCAIL-2) |
| Inference FPS | 13.6 (3 steps) | 1.367 (Wan-Animate, 20 steps) |
On identity, UniSwap (0.629) merely matches SCAIL-2 (0.630). Audio quality SIG at 3.486 ties Seed-VC at 3.489, but speaker similarity SECS is only 0.730, well below Seed-VC's 0.829, so the learned timbre is less faithful.
Long video (1 minute, three segments): UniSwap's DINO-S holds at 0.596 / 0.590 / 0.596 and IQA between 3.966 and 4.032, while SCAIL-2's DINO-S falls from 0.566 to 0.517.
Efficiency: 241 frames on one H100, UniSwap does 3 steps, 1.76 s per block, 13.6 FPS, about 10x faster than the fastest baseline and about 100x faster than MoCha.
User study (30 participants): top scores on appearance identity (4.16) and lip sync (4.11).
Putting face- and voice-swapping into one model that also streams is a first for this task. A single model means lip motion and timbre share a joint constraint, so sync is steadier, and streaming enables real-time interaction like live character replacement. 13.6 FPS on a single card is usable in practice. But identity and timbre fidelity only match the strongest cascade rather than beating it, so the selling point is joint plus streaming, not raw quality.
The authors state that the system handles only single-speaker talking videos; multi-speaker scenes, occlusions, and complex interactions remain out of reach, and facial expressions are driven automatically by audio rather than edited independently.
One gap stands out: speaker similarity SECS at 0.730 is a real step behind Seed-VC's 0.829, which makes voice transfer the relatively weak link. The paper does not test generalization across more speakers or accents, and the user study covers only 30 people.