Tencent's WeMM-Video: Open Omni-Modal Models Where a 9B Beats Qwen3-Omni-30B

WeMM-Video: Pushing the Frontiers of Open-Source Omni-Modal Video Understanding

Changli Tang, Jie Yang, Tianyi Wang, Lei Li, Daoze Zhang, Zijie Xin, Zhan Liu, Li Li, Jinchang Xu, Fengyun Rao, Chao Zhang, Jing Lyu

cs.CV, eess.AS

2026-11-20

Tencent's WeChat team open-sources four omni-modal models (2B to 35B-A3B), trained on 1.2T tokens, leading open models on 33 audio-visual benchmarks.

What problem this solves

Open-source models still handle video with sound poorly. The report lists three recurring failures: they miss key audio-visual details, they fail to tie a sound to the visual event that caused it, and they hallucinate across modalities, stating things neither the picture nor the audio supports. Live-stream commerce and short-drama analysis, which the team targets, depend heavily on exactly this joint reasoning.

Data is the other obstacle. Verifiable audio-visual reasoning data is rare in the wild, and the base LLMs have no hearing at all. The goal here is a full family at once: 2B, 4B, 9B and 35B-A3B, Chinese-English, covering text, image, audio and video, with hearing added without degrading pure vision or text.

Method

The design follows video-SALMONN 2. The 2B/4B/9B models sit on Qwen3.5-2B/4B/9B; the 35B-A3B sits on Qwen3.6-35B-A3B. The visual encoder is kept as is. Audio goes through a Whisper-style encoder (initialized from Qwen2.5-Omni) and a window-level Q-Former, a small module that compresses features with learned queries. Audio is sampled at 16 kHz, cut into 30-second chunks, and each 0.1-second window becomes one token, i.e. 10 audio tokens per second. For video with sound, visual and audio tokens are interleaved chronologically with textual timestamps so the model can align them itself.

The corpus is about 1.2T tokens. Joint audio-visual data is 45.4%; captioning is the largest task at 37.7%. The team built action-dense visual captions (sports, esports, dance), audio-visual captions from films, short dramas, interviews and live commerce, plus cross-media tasks such as video comparison, soundtrack matching and product-to-ad consistency checks.

Pre-training has three stages:

Post-training also has three stages:

Results

Evaluation spans 120+ benchmarks: 33 audio-visual, 32 silent-video, 18 audio, 28 image, 12 text.

ComparisonScopeResult
35B-A3B (Thinking) vs Seed2.0-Mini-042817 general audio-visual benchmarkswins 16
35B-A3B (Thinking) vs Qwen3.5-Omni-Flashsame 17wins 16
9B (Thinking) vs Qwen3-Omni-30B-A3B-Instruct, MiniCPM-o-4.5-9Bmost general audio-visual benchmarksahead on most
4B (Thinking) vs 30B-class baselinevast majority of benchmarksahead
9B (Thinking), Video-MME w/ audiovs Gemini-3.0-Flash85.1 vs 88.6
9B (Thinking), Video-MME-v2 w/ audiovs Gemini-3.0-Flash51.6 vs 63.2

The gap to Gemini-3.0-Flash widens on the harder v2 benchmark. The paper says "approaching" Gemini, not beating it. Thinking mode beats instruct mode on most audio-visual comprehension and reasoning tasks, though the paper describes the gain in words rather than a headline number.

Other modalities are mostly parity. On silent video, the 35B-A3B matches its own backbone Qwen3.6-35B-A3B and Seed2.0-Mini-0428, and the 9B matches or edges Qwen3.5-9B on most benchmarks. Image and text results also match the backbones, with some gain in instruction following. On audio, the 35B-A3B beats open baselines on MMAU-Pro, MMAR and MMSU and is on par with Qwen3.5-Omni-Flash.

Why it matters

For teams building audio-visual applications, this is one of the few open omni-modal families that ships weights plus the full evaluation and reproduction pipeline, in four sizes. A 9B model beating a 30B Qwen3-Omni is a real deployment saving.

The contribution is mostly an engineering recipe: audio alignment, three-stage post-training, repetition control. The architecture extends an existing line. This is solid incremental work.

Limitations

Terms

Source

What people are saying

Related papers

All paper explainers