VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation
Yinming Huang, Shuyuan Tu, Xi Yan, Zihan Yang, Jianhua Han, Xu Hang, Yu-Gang Jiang, Zuxuan Wu
cs.CV
2026-08-19
VA-Judger, trained on 10.3K human AV pairs, hits 68.43% pairwise accuracy; post-training LTX-2 with it wins 62.3% of three-way human votes versus 27.6% for OmniNFT.
Joint video-audio generators now emit a picture track and a synced soundtrack in one forward pass. Reinforcement learning is the obvious next step, and it needs a reward. The current recipe, OmniNFT, concatenates off-the-shelf scores: VideoAlign and HPSv3 for visuals, Audiobox Aesthetics for sound, CLAP for audio-text match, DeSync and SynchFormer for timing.
People do not score a clip that way. A high sync number can still narrate the wrong event. A pretty frame can sit under the wrong mood. Several metrics can all fire while the clip still feels incoherent. Turn those single-axis scores into pairwise winners and they match humans only 50.43% to 56.88% of the time; no representative metric clears 56.88%. Optimizing them invites reward hacking: the dashboard rises, the clip still looks fake. Image reward models never hear the audio. Audio metrics never see the picture. Adding the two scores does not recover the cross-modal call.
The group at Fudan University, Shanghai Innovation Institution, and Yinwang Intelligent Technology builds three pieces: a preference set, a judge, and a post-training loop that uses the judge.
VAPref-10K starts from 10,173 real clips from YouTube, Bilibili, films, and TV series, trimmed to 5-10 seconds and sorted into six buckets: multi-person dialogue, voice-over, music, cinematic ambience, environmental events, and talks. Qwen3.5-Omni captions them with dense timestamps; a Qwen rewrite turns those captions into generation-ready prompts, leaving 10,083. Open-source generators OVI, LTX-2, and DaVinci-MagiHuman produce the candidates. Annotators never assign absolute scores. They pick the better clip of a same-prompt pair and tick the quality dimensions that justify the pick.
Pairs are split by difficulty. The easy pool has 4.4K comparisons, mostly OVI versus LTX-2, plus DaVinci-MagiHuman on speech-heavy categories, where the quality gap is obvious. The raw hard pool has 9.8K pairs, mostly two seeds of the same model on the same prompt. Headline stats: about 9K prompts and 10.3K fine-grained pairwise labels.
VA-Judger starts from Qwen3-Omni-30B-A3B-Instruct. Visual and audio encoders stay frozen; only the language backbone trains. The model sees a fixed rubric, a text prompt, and two clips with audio. It scores each clip 1-10 on five axes with a short reason, sums the scores, and closes with <answer>video k is better</answer>. The five axes are (A) prompt alignment, (B) video-audio consistency, (C) audio quality, (D) video quality, (E) completeness and coherence.
Training has three stages.
VA-Judger-Bench holds 1,150 held-out pairs: 400 easy, 250 in-domain hard, 500 out-of-domain. The OOD split, built from AVGenBench, pits Kling 2.6, Wan 2.6, Veo 3.1 Fast, Veo 3.1 Quality, and Sora 2 against each other. None of those closed models appear in training.
For generation post-training they keep OmniNFT's RL loop on 19B LTX-2 and swap the five expert rewards for VA-Judger. Eight candidates per prompt make 28 pairs. Dimension C becomes the audio reward, D the video reward, and the mean of A, B, and E a shared cross-modal reward. The backbone stays frozen; only LoRA trains.
Judge accuracy first.
| Method | Easy | In-domain | OOD | Overall |
| Javis Score (best single metric) | 60.00 | 55.51 | 54.96 | 56.88 |
| Qwen3-Omni Instruct CoT | 63.25 | 54.80 | 55.00 | 57.83 |
| Easy SFT | 72.00 | 59.20 | 56.20 | 62.35 |
| Hard SFT | 74.50 | 63.60 | 60.20 | 65.91 |
| GRPO (VA-Judger) | 76.25 | 66.00 | 63.40 | 68.43 |
Single-dimension metrics sit between 50.43% and 56.88% overall. VideoAlign and AudioBox look usable on easy pairs (62.39% and 58.70%) and collapse out of domain (48.35% and 44.00%). Each training stage adds points. The final model is 10.60 points above Qwen3-Omni CoT.
Post-training LTX-2 on a random 200-prompt JavisBench slice:
| Method | VQ | MQ | AQ | JavisScore | AVHScore | Text-audio align |
| LTX-2 | 2.248 | 0.697 | 4.767 | 0.074 | 0.091 | 0.105 |
| OmniNFT | 3.727 | 0.947 | 5.399 | 0.122 | 0.146 | 0.133 |
| VA-Judger | 3.942 | 1.183 | 5.610 | 0.230 | 0.261 | 0.180 |
VA-Judger ranks first on 11 of 13 JavisBench metrics and 6 of 7 extra metrics. DeSync (lower is better) goes the other way: 0.430 for the base model, 0.226 for OmniNFT, 0.592 after VA-Judger. OmniNFT paid for the sync metric. Humans did not.
Twenty viewers ran a three-way forced choice on those 200 prompts: 62.30% picked VA-Judger post-training, 27.63% OmniNFT, 10.08% base LTX-2. Roughly twice OmniNFT and six times the base model.
RL for joint AV generation has been starved of a reward that actually watches the full text-video-audio event. This paper swaps stacked public metrics for pairwise human labels plus an inspectable rubric, then shows the swap is visible to people, not only to JavisScore.
Teams already running an OmniNFT-style loop can drop the judge in as the reward head. The code is public. Treat it as a working reward, not as a general AV evaluator: 63.40% OOD accuracy still misses a large fraction of close calls among closed models.
The novelty is the combination (preference data, easy-to-hard SFT, GRPO split onto human-ticked dimensions), not a new architecture. The claim that the reward has to see all three streams at once is now hard to dodge.
The paper never writes a limitations section. Several gaps sit in the numbers.
OOD accuracy is 63.40%. Better than chance, not a reliable referee. Easy-stage labels come from Gemini (80% in a pilot). Hard-stage rationales are still Gemini text, filtered only so the final winner matches the human. The model may learn Gemini's reasons for a human choice.
Temporal sampling is thin: at most 12 frames and 128 tokens per clip. Lip-sync and event order are asked to survive that. DeSync getting worse is consistent with the judge trading frame-level timing for semantics and picture quality. The authors call OmniNFT's DeSync gain reward hacking. They never test whether VA-Judger hacks a different proxy.
Training generators are only OVI, LTX-2, and DaVinci-MagiHuman. Post-training only LoRA-tunes LTX-2. The 20-person three-way study reports no inter-annotator agreement. Dataset copy also disagrees with itself: the intro says seven categories and three models, the contributions say eight generators, the method says six categories. Eight likely counts the closed models used only at test time.