CUHK and Alibaba propose OmniVChat: native audio-visual dialogue trained on synthetic data
机器之心 · wechat · 2026-09-28
Researchers from CUHK, Alibaba TokenHub, SJTU and others introduce OmniVChat, defining native audio-visual dialogue where a model directly consumes user audio and video simultaneously — no text prompts, subtitles, or ASR needed — preserving tone, expressions, and camera orientation.
To tackle scarce real dialogue data and missing evaluation standards, they take a 'generation for comprehension' route with three components: a multi-agent data engine (Studio) synthesizing single/multi-turn dialogues with reference replies and tiered rubrics; a 2,800-case benchmark (Bench) covering connection awareness, entity alignment, self-awareness, anti-hallucination, and emotion recognition; and an RL scheme (OmniChat-RL) turning rubrics directly into rewards.
Built on Qwen3-Omni-30B-A3B-Instruct, the key finding: RL on purely synthetic data lifted rubric scores from 0.465 to 0.652 on Bench, and from 0.402 to 0.632 on a human-recorded set never seen in training — synthetic training generalizes to real interactions. Paper, dataset, and code are open-sourced.
More from Research
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- Kind ICLR Reviewer Raises Score Despite Unresolved Concerns, Citing the Paper's Value — MarkSchmidtUBC · 2026-11-16
- Hand-editing 0.2% of weights plus 100 CPU-minutes turns an English TTS model into an 18MB Sanskrit one — kastnerkyle · 2026-11-16
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16