SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model
Aviad Dahan, Rajaei Khatib, Yonatan Bitton, Idan Szpektor, Lior Wolf, Raja Giryes
cs.SD, cs.CV
2026-10-08
SepGen trains a 327M audio LoRA on frozen LTX-2.5. One sample emits video, the mix, and two captioned stems. Dialogue WER is 0.07 versus 0.81+ for cascade separators.
Joint audio-video generators can already match a picture with a soundtrack. LTX-2, Ovi, and Veo 3.1 Lite still emit that soundtrack as one mix. The sources inside it cannot be moved or brought closer on their own.
Once a monocular video is lifted to a 4D scene and rerendered from a new camera, spatial audio needs a waveform per source, so each source can be placed in the scene and carried to the listener. A mix has no such handle. The sound stays locked to the original viewpoint.
A language-queried separator run afterward does not fix identity. Each caption is its own pass. Asked separately for two men talking, both queries can return the louder voice. The stems have to be predicted together, off the same mix.
The backbone is frozen LTX-2.5 22B, a flow-matching model: a velocity field pushes noise back toward data, with one transformer stream for video and one for audio. Training updates only an audio-stream LoRA, rank 128, 327M parameters, on a single A6000, with the backbone in int8. Separation takes 12,000 updates. Generation continues for 3,000 more.
Three equal-length audio channels carry two stems, one waveform per source, plus the mix. They share the backbone's temporal grid and rotary positions. Stems attend to each other and to the mix. The mix attends only to itself, and the LoRA is switched off on mix tokens. Neither video nor mix reads the stems. Turning the adapter on and off from the same input leaves mix activations and video bit-identical at every layer, so the adapter does not rewrite the backbone's picture or soundtrack.
Each stem reads only its own caption. Cross-stem attention guidance, at scale 2, runs a positive branch on that caption and a negative branch on the sibling caption, pushing a sound off the wrong stem when both captions fit.
The training set is 4,843 two-source clips: 1,658 CelebV-HQ talking-head pairs, 1,437 URMP instrument duets, 749 generated pairs, and 999 composites of VGGSound and MUSIC-21. Each clip has up to three caption styles, drawn at random. The loss hits stem velocity only. Video and mix are conditions.
Stage 1 always shows a clean mix, and the stems learn to take it apart. Stage 2 keeps the mix clean on 30 percent of samples so separation survives. Otherwise the mix is noised, but never past the stem noise. That covers the two states generation actually meets: a mix as noisy as the stems, and a clean estimate that already exists.
Separation locks the observed video and mix at zero noise and denoises only the stems. Generation denoises all three together. Above noise 0.97 the stems read the noisy mix. Below it they read the clean mix implied by the current velocity. If they watch a mix that has not settled for the whole trajectory, they write content the final mix never contains.
The generation set has 194 prompts, 134 sound scenes and 60 dialogues. SepGen emits video, mix, and both stems in one sample. FlowSep, AudioSep, and SAM Audio then separate that same mix with the same captions. This board has no reference waveforms. Judge scores a stem from 1 to 5. CLAP, a contrastive text-audio similarity, still rewards a copied mix, while J-swap falls to 0 on that copy, so the headline number is J-swap: Judge on the stem's own caption minus Judge on the other caption. Speech uses Whisper word error against the speaker's own line, and the cosine of WavLM speaker embeddings between the two stems. A lower cosine means the voices are less alike.
| Method | Judge↑ | J-swap↑ | WER↓ | Speaker cosine↓ |
| FlowSep | 3.35 | 1.26 | 1.03 | 0.97 |
| AudioSep | 3.23 | 1.27 | 0.91 | 0.87 |
| SAM Audio large | 2.93 | 0.42 | 0.81 | 0.95 |
| SepGen, joint | 3.82 | 2.07 | 0.07 | 0.66 |
| SepGen, separate again | 4.21 | 2.29 | 0.10 | 0.67 |
CLAP does not put SepGen reliably ahead of FlowSep or AudioSep. AudioSep's query encoder is the CLAP checkpoint used for scoring. J-swap is where the gap opens. Joint sampling binds each stem to its own caption. A separate query can return the same source twice. A separation-only checkpoint used as a generator sits at J-swap 1.95 and WER 0.10, with a seed spread of 0.13 on J-swap. The short continuation moves those to 2.07, 0.07, and 0.01. Separating the generated scene again raises the sound scores. Its WER of 0.10 and the joint run's 0.07 sit inside seed variation.
On 149 scenes from Veo 3.1 Lite, the separation checkpoint scores Judge 4.36 and J-swap 2.61, against 3.46 and 1.60 for AudioSep. With the spoken lines left in the captions, WER is 0.05, against 0.74 to 0.98 for the cascades. Strip the lines out and SepGen rises to 0.50, while the cascades stay between 0.73 and 1.03. Four or five of the 90 speech stems go silent, and part of the drop in speaker cosine comes from that silence. SAM Audio draws eight candidates and keeps the best by Judge and CLAP. Under that same reranker on Veo, SepGen's Judge is 4.49 against 4.10 and 4.15, and WER stays at 0.05 against about 0.55.
On 172 real recordings, Judge comes from the same encoder family as SAM Audio. SepGen scores 3.70 against 3.12 for SAM large-tv, and J-swap 1.42 against 0.78. With the benchmark's original wording, SAM large-tv rises to 3.32 and is still behind.
Listeners could tie. Against SAM Audio's best of eight, 48 people gave SepGen 0.58 of separation judgements and SAM Audio 0.25. Real recordings were 0.64 to 0.20. Veo was 0.51 to 0.30. Against AudioSep, 51 people gave generation 0.63 to 0.28. Dialogue was 0.87 to 0.10. Sounds were 0.49 to 0.38.
On 129 held-out scenes with true stems, envelope correlation is 0.82 and log-spectral distance is 6.3 dB, against 0.76 and 11.0 for AudioSep. On 57 dialogues, WER is 0.05, the same as the true stems, while the strongest cascade is 0.29. A silent mix drops Judge to 1.10. A mix borrowed from another clip makes the transcript follow the donor, with donor-line error 0.41 and original-line error 1.16. Sampling each caption alone drops generation Judge from 3.82 to 2.27. Removing the video does not move separation scores by a significant margin. Retraining with the timeline laid end to end, instead of aligned in time, more than triples WER.
An existing audio-video generator can grow one waveform per source without a new 22B training run and without a separator chained on after generation. The adapter is 327M parameters and fits on one GPU. The mix channel is protected, so the original picture and soundtrack stay. One set of weights separates when the mix is clean and generates when all three latents are denoised together.
The gain sits on dialogue, where two sources belong to the same kind. The listening margin on sounds, 0.49 to 0.38, is measurable and modest. Binaural audio at a new camera is still someone else's renderer. The waveforms that come out still carry the original room.
The layout is fixed at two sources. When both stems are the same instrument and the captions match, log-spectral distance rises from 6.3 dB on distinct instruments to 11.7 dB, and a permutation-invariant distance is still 11.4 dB. The text cannot aim the cut.
The decoder rewrites phase and keeps envelope and spectrum. SI-SDR against a reference mostly measures that phase. Stems also keep the original reverberation and distance cues. They are not dry tracks a spatial renderer can drop onto a new listener.
The new-viewpoint soundtrack in the overview figure is an illustration. The evaluation never closes that loop, reports no localization error, and never plugs the stems into a renderer. Separation scores barely move on unrelated frames. Seeing the sounding object is not where these numbers come from.
Most of the leaderboard has no reference stems. The listening study uses ten clips. On Veo, majority vote by source is 3 wins and 4 losses for SepGen, while the judgement share is 0.51 to 0.30. The 95 percent intervals on the sounds listening test overlap. WER is 0.05 when the caption contains the spoken line and 0.50 when that line is removed. SepFormer, a dedicated speech separator, matches the separation checkpoint only after the lines are gone. A real share of the dialogue lead is the script already sitting in the caption.