OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
Haolin He, Yunfei Chu, Qi Chen, Wen Huang, Yuan Feng, Muzhi Zhu, Zheqi Dai, Haoning Xu, Dongchao Yang, Chunyat Wu, Zining Liang, Zhengxi Liu, Xiquan Li, Xie Chen, Xize Cheng, Qize Yang, Jin Xu, Qiuqiang Kong
eess.AS, cs.AI, eess.IV
2026-09-18
CUHK and Alibaba GSPO-tune Qwen3-Omni on synthetic native AV chats: Mean 0.465 to 0.652, 360 real Chinese phone videos 0.402 to 0.632, replies 79 to 36 words.
Omni models can take audio and video at the same time, but public recordings of people talking to a camera on their own phones, walking, with real noise, are scarce. Most stacks still run ASR, then hand the transcript to a vision model. Prosody, pauses, and affect drop out. Latency and transcription error come in.
Scoring is the other bottleneck. A good reply may need the pattern on a mug, left versus right in the current camera frame, or a worried face. The same fact can be phrased many ways, so keyword match is a poor judge. CUHK, Alibaba Token Hub, and Shanghai Jiao Tong frame this as OmniVChat: the user's query lives in the audio-visual stream, the model answers in text, with no separate typed question, no captions, and no ASR cascade.
Three pieces share one task definition: a data engine, a benchmark, and an RL reward.
OmniVChat-Studio is a four-agent pipeline. The Director plans and reviews text. The Renderer turns an accepted script into a synced clip (1080P, 44.1 kHz stereo). The Reviewer writes a caption and a seven-part quality report. The Validator applies deterministic rules. A single-turn run samples attributes, writes a timed script, renders, then writes a reference reply and a tiered rubric against what the clip actually shows. Multi-turn runs keep accepted earlier clips as history; later turns cite prior video, audio, or last frames. Attributes are drawn from a config: language mix, 22 scenario domains, 14 camera setups, noise levels. Repairs do not resample attributes, so a hard case cannot quietly become an easy one. Each repair route has a budget; exhausting it discards the run.
Scoring uses a gated rubric. Tier 0 checks that the reply is in the user's language. It is worth zero points, and failing it zeros the item. Later tiers only pay after every earlier criterion is met. qwen3.7-max is the grader. The item score is credited points over non-gate criteria. The headline metric, Subcategory Mean, averages inside each of 17 subcategories, then averages those 17 equally.
Training fine-tunes the Thinker of Qwen3-Omni-30B-A3B-Instruct with GSPO and rank-64 LoRA. GSPO uses a length-normalized sequence likelihood ratio, which is more stable on MoE models. Each prompt samples four replies. Advantages subtract the group mean, without standard-deviation scaling. Total reward is rubric score (weight 1) plus format 0.5, efficiency 0.1, and style 0.5. Efficiency is a min-max of "score per counted word" inside the group, so there is no global length target. Style is a 0/1 from qwen3.7-max on seven spoken-style checks. Training runs 1,000 steps of 32 prompts; step 940 is picked on the development split.
The bench has 2,800 synthetic dialogues (2,550 single-turn, 250 multi-turn), about 37% Chinese and 63% English, across five abilities, 17 subcategories, and 22 domains. Training uses 5,600 further synthetic dialogues plus a 560-item dev split. OmniVChat-Bench-Human is 360 Chinese phone recordings, 30 per each of 12 single-turn subcategories, held out of training.
| System | Synth Mean | Human | Credit / 1k words | Style | Length |
| Qwen3-Omni-Instruct | 0.465 | 0.402 | 5.75 | 0.710 | 79 words |
| OmniVChat-RL | 0.652 | 0.632 | 18.38 | 0.992 | 36 words |
| Gemini-3.5-Flash | 0.667 | 0.509 | 10.59 | 0.755 | unreported |
| Gemini-3.7-Flash | 0.640 | 0.575 | 16.96 | 0.631 | unreported |
During training, the rubric term moves from 0.308 to 0.629, style from 0.750 to 0.992, and mean length from 91.5 to 36.6 words. The Chinese synthetic slice goes 0.437 to 0.634; the recorded set goes 0.402 to 0.632. The two curves track.
Dropping the efficiency term lifts Mean to 0.697 and Human to 0.691, but replies grow to 99 words and credit per thousand words falls to 7.12. Dropping style leaves Human at 0.632 and cuts Style from 0.992 to 0.788. The full reward is a trade among correctness, brevity, and spoken tone. Gemini-3.5-Flash still leads synthetic Mean at 0.667. On recordings, OmniVChat-RL at 0.632 beats Gemini-3.7-Flash at 0.575. Synthetic rank does not dictate recorded rank.
By ability, anti-hallucination jumps 0.186 to 0.667 and self-awareness 0.388 to 0.734. Emotion recognition falls 0.694 to 0.651: shorter replies cost some affect. Dialogue-state and camera pose stay weak at 0.573, against 0.623 for Gemini-3.5-Flash.
Generation-for-comprehension works far enough to train and to test, and the gains show up on real phone videos. For people building voice-and-camera assistants, gated rubrics fit open replies better than keyword match. The efficiency term cuts answers from 79 words to 36, which users notice.
This is not a closed-source replacement. Gemini-3.5-Flash still leads the synthetic Mean. The lift is a LoRA on Qwen3-Omni; a new backbone needs a rerun.
The human probe is Chinese and single-turn. It does not test multi-turn dialogue, live barge-in, or latency. Multi-turn scoring grades only the last reply, with earlier turns fed as reference answers. Close synthetic and recorded scores do not prove matched distributions.
Grading is all LLM. Six judges on the same 2,773 Gemini replies differ by 0.034 in pooled score; Cohen's κ runs 0.532 to 0.845. The paper does not remeasure the 0.465 to 0.652 gain with another judge. The Renderer is never named as a specific video model. Code and data are promised, not shipped at submission.
Rendered clips grow malformed arms. The engine reseeds and rewrites prompts; humans then inspect. Generation quality still caps the tail. JoyAI-VL-Interaction with Qwen3-ASR is the one cascade baseline, and it trails native Qwen3-Omni. One cascade is a thin comparison.