OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning
Haibo Wang, Jiteng Mu, Jialu Li, Jingru Yi, Yuanjun Xiong, Jianming Zhang, Lifu Huang, Mingze Xu
cs.CV
2026-10-02
Adobe and UC Davis turn Qwen3-Omni-30B into a multi-turn agent that fetches raw audio/video clips on demand before answering; it leads open-source models on 10 audio-visual benchmarks.
Omni-LLMs, foundation models that take text, audio, and video together (Qwen3-Omni, Nemotron-Omni), encode the whole stream in a single forward pass. On long videos that backfires: a two-second sound or a small visual detail gets diluted in minutes of irrelevant content, and the model falls back on language priors or answers from one modality. Prior fixes stay single-modality. DeepEyes and Pixel Reasoner taught models to think with images; Video-o3 and VideoZoomer added temporal retrieval tools for video. Audio is either absent or processed in lockstep with video. OmniSeek, from UC Davis and Adobe Research, makes evidence acquisition part of reasoning itself, with the two modalities decoupled.
The core is an iterative <think> → <toolcall> → <observe> loop. The model starts with a coarsely sampled global context, then each turn decides whether to look or listen, and over which time window:
Retrieved segments go back into the context as raw audio or video tokens, never as text. Termination is not hardcoded: the model judges in <think> whether the accumulated evidence suffices, and either plans the next call or emits <answer>. Decoupled tools let it grab an audio cue first, then pull a high-resolution clip from an entirely different part of the video.
Training runs in three phases:
That reward closes a specific hole in outcome-only RL: a trajectory answered correctly using one modality earns the same credit as one that truly needs both. The method runs two extra teacher-forced forward passes on the already-generated trajectory, zeroing the attention keys of audio tokens in one pass and visual tokens in the other, and measures how much each generated token's log-likelihood drops. The two per-modality necessity scores combine through a min operator, a logical AND gate where the weaker modality sets the reward, applied only to correct answers. Cost: two nograd forward passes, no extra rollouts.
Built on Qwen3-Omni-30B-A3B-Instruct, evaluated on 10 omni-modal benchmarks:
| Benchmark | Base | OmniSeek |
| MMOU | 54.1 | 70.4 |
| LVOmni | 35.8 | 44.2 |
| Daily-Omni | 71.9 | 80.0 |
| VideoHolmes | 59.1 | 74.6 |
| Video-MME | 76.8 | 78.5 |
Gaps widen as videos get longer and evidence gets sparser: +16.3 on MMOU, +8.4 on LVOmni. On VideoHolmes, OmniSeek reaches 74.6 against 62.9 for OmniVideo-R1, a text-only reasoning model; on OmniVideoBench it is 47.7 vs 44.8. Against a baseline given the same RL but restricted to pure textual thinking, multi-turn tool use adds +13.5 on OmniVideoTest and +8.1 on WorldSense. The paper also reports beating Gemini-2.5-Pro on FutureOmni, VideoHolmes, and JointAV.
Ablations separate each contribution. Phase-1 SFT alone hurts (Daily-Omni drops from 71.9 to 69.1, a classic alignment tax); Phase-2 RL recovers it to 75.5; G=16 raises it to 78.2; the necessity reward brings 80.0, contributing +3.2 on WorldSense and +2.2 on OmniVideoTest on its own. Tool-call counts track task demands: JointAVBench peaks at 3 calls (80.8% vs 68.1% at 2), Daily-Omni at 4 (87.1%), while VideoHolmes and MMOU need 5 or more (75.9%, 73.3%). General video ability survives training: 78.5 on Video-MME and 77.1 on MLVU, ahead of 72B Qwen2.5-VL and LLaVA-OneVision.
This is the agentic retrieval lineage of DeepEyes and Video-o3 extended systematically to decoupled audio plus video, with modality routing itself now a learned decision. Two pieces are reusable anywhere: the data engine's deterministic construction of actions and observations is a cheap fix for trajectory hallucination, and the attention-masking dependence measure with a min gate fits any multimodal RL setup that needs to suppress single-modality shortcuts. For long-video applications, on-demand clip retrieval is a third road beside longer contexts and token compression. The mechanism itself is a transplant from single-modality predecessors; the real increments are the data engine and the necessity reward.