AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation
Zhiyu Xu, Weilong Yan, Yufei Shi, Shiyang Li, Yihao Liu, Kin-Man Lam, Yuewen Cao
cs.CV, cs.SD
2026-09-24
AV-GRPO freezes one modality trajectory and updates the other. Full FT of LTX-2.3 22B lifts audio quality 5.097 to 5.798 and cuts DeSync 0.757 to 0.607.
Joint audio-video generators already denoise a picture and a soundtrack in one pass. Dual-tower models such as LTX-2 and JavisDiT exchange information through cross-modal attention. Quality, text alignment, and synchronization still come apart: a watchable clip with unsynced lips, or two decent streams whose events do not match.
Dropping Flow-GRPO onto paired rollouts hits three walls. Audio quality, visual quality, text alignment, and sync can vote for different candidates, so one group-relative advantage cannot say which tower earned the reward. Sync scores also depend on the counterpart stream: matching audio to a still shot is easier than matching frequent on-screen events, so within-group comparisons are not fair. Both towers must be backpropagated together, which is tight at 22B, and the two streams do not share learning speed or noise scale.
AV-GRPO splits the joint problem into two conditional subproblems and alternates. In an audio-anchored phase it first samples a full audio-video pair, locks the entire audio denoising trajectory, draws G independent video trajectories, and updates only the video tower. The video-anchored phase swaps roles. The active tower switches every n steps; the main runs use n=4.
Locking the full trajectory matters. The towers talk through cross-modal attention at every denoising step, so freezing only the final audio frame still lets intermediate states drift. Every video sample in a group faces the same audio, which controls sync difficulty. The anchor reward is constant inside the group and vanishes after centering, leaving target-modality quality and alignment plus sync against one shared counterpart.
Freezing the other tower is also an accounting move: the anchor still has to be sampled, but gradients and optimizer state live only on the target. Shanghai AI Lab used that cut to full-parameter post-train 22B LTX-2.3 on eight A800 80GB GPUs.
Hyperparameters are split on purpose. Video noise strength aV=0.02, audio aA=0.8. The video tower also reweights the policy and KL terms across timesteps, following LongCat-Video, so late steps that set brightness and fine detail are not starved. Sampling uses 15 steps. Near t=1 the diffusion coefficient blows up; Adaptive Noise Clipping caps the standard deviation of the stochastic increment at a threshold τ.
Rewards are split too. Video: VideoAlign, CLIP, DeSync. Audio: Audiobox Aesthetics, CLAP, DeSync. Lower DeSync is better, so the sign is flipped before group standardization. Audio adds a CLAP guardrail: when group-mean CLAP sits below a threshold, only the alignment advantage is sent back, blocking a shortcut that trades prompt fidelity for louder quality and sync.
The training set, 5DAV, is cut along five orthogonal axes: semantic hierarchy, sound-source type, sync difficulty, temporal complexity, and instruction granularity. 3×4×4×2×3=288 bins, 20 prompts each, 5,760 total. Sampling ratios can be retuned by stage. Generation is 544×960, 97 frames, 24 FPS.
The main comparisons are LTX-2.3 22B and GDPO under the same hyperparameters and data order. LoRA and full fine-tuning each take 480 effective update steps. Because AV-GRPO alternates towers, the logged total is 1,440, so each tower sees the 5,760 prompts once.
| Method | AQ↑ | CLIP↑ | CLAP↑ | AV-IB↑ | JavisScore↑ | DeSync↓ |
| LTX-2.3 | 5.097 | 0.318 | 0.408 | 0.212 | 0.183 | 0.757 |
| +GDPO | 5.406 | 0.313 | 0.431 | 0.224 | 0.202 | 0.708 |
| +AV-GRPO LoRA | 5.633 | 0.330 | 0.449 | 0.236 | 0.213 | 0.554 |
| +AV-GRPO full | 5.798 | 0.327 | 0.468 | 0.247 | 0.222 | 0.607 |
Full fine-tuning lifts JavisBench audio quality from 5.097 to 5.798, CLAP from 0.408 to 0.468, AV-IB from 0.212 to 0.247. Against GDPO, AV-IB moves 0.224 to 0.247, JavisScore 0.202 to 0.222, DeSync 0.708 to 0.607. LoRA posts a lower DeSync (0.554) and a slightly higher CLIP (0.330 vs 0.327), but VQ falls from 5.855 to 5.816. Visual quality barely moves. The gains sit in audio, alignment, and sync.
VABench follows the same pattern. Full fine-tuning takes audio aesthetics 3.319 to 3.631, lip sync 1.351 to 1.646, DeSync 0.800 to 0.542. Against GDPO, lip sync 1.439 to 1.646 and DeSync 0.726 to 0.542. Visual realism is 4.395 versus 4.399 for the base model, a wash slightly down. Ablations on JavisBench-mini: 5DAV beats VGGSound; dropping the easiest or hardest sync/instruction bins hurts sync metrics; no alternation, or switching every 2 or 8 steps, loses to a 4-step schedule. Those ablations are radar charts and relative changes, not a numeric table.
The hard part of RL post-training for joint generation is credit assignment, not another reward model. AV-GRPO's move is to stop updating both towers from one shared advantage, and instead freeze the counterpart's full trajectory so the target tower competes under a controlled condition. Frozen-tower training puts 22B full-parameter post-training on eight GPUs, a scale a lab without a giant cluster can actually run.
This is an engineering decoupling, not a new generator. The backbone is still LTX-2.3; the rewards are off-the-shelf VideoAlign, CLIP, CLAP, and DeSync. Teams already running Flow-GRPO on dual-tower flow models should steal the anchored rollouts and per-tower hyperparameters. Unimodal video or Foley has less to gain.
There is no standalone Limitations section. Several cracks are visible anyway. Visual quality barely moves, and LoRA VQ even dips, so the method is closer to repairing audio and sync than raising the generation ceiling. GDPO is counted at 480 steps with both towers updated at once; AV-GRPO logs 1,440 total steps, so the compute bill is not symmetric. Ablations do not publish citable absolute numbers; the claims about switch interval and 5DAV rest on figures. Every reward is a proxy. The authors themselves catch an audio shortcut that drops CLAP to boost AQ and DeSync, which is why the guardrail exists; how the threshold is chosen, and whether another hack replaces it, is not audited further. The joint-optimality appendix assumes exact Gibbs kernels and Dobrushin contraction. A 15-step SDE sampler sits some distance from that. The study covers one model, LTX-2.3 22B.