Omni-Streaming Thinking
Enjun Du, Siyi Liu, Ziyu Zheng, Jingyu Li, Yiwen Guo, Yongqi Zhang, Difan Zou
cs.LG
2026-09-14
With a frozen Qwen3-Omni backbone, Omni-Streaming Thinking lifts SOVBench to 87.8 vs StreamOV’s 81.6 and raises diagnostic d′ to 2.95 from at most 1.38.
A streaming omni-modal model has to answer from the video chunks and audio seen so far. Visual evidence often matures first: a departure board already says Gate 6 while the announcement is still unfinished. If that reading is written into memory as a fact, later reasoning keeps repeating it after the audio has moved the flight to Gate 12.
The paper names this failure premature cross-modal commitment. Existing streaming systems compress tokens, retrieve history, or decide when to speak. They rarely attach an unfinished guess to a future test on a specified modality. OST-DiagBench holds the video fixed and edits only the soundtrack, so agreement no longer hides visual bias.
OST freezes Qwen3-Omni-30B-A3B-Instruct and adds LoRA plus three light heads: a forecaster, a verifier, and an answer gate. Perception runs every second; a decision chunk lasts 4 s. Each chunk writes a six-field Omni-State covering visual evidence, audio presence, audio evidence, conflict, forecast, and sufficiency.
A claim records a predicted future observation, which modality can test it, a deadline, and which later states depend on it. Claims start pending. Audio and visual tokens have separate budgets, and pending evidence windows are pinned so dense frames cannot evict the waveform needed for the test. Long-term memory merges 4 s states into 16 s and 64 s summaries, then a long-term root.
When the window closes, the verifier returns CONFIRMED, REFUTED, or UNRESOLVED. A refutation lowers reliability along a provenance graph; decoding adds log ρ to attention logits over the claim and its descendants. Verdict-conditioned rewriting then contrasts logits with and without this chunk’s refutations, and generates a corrected state. The answer gate waits until evidence is sufficient and no answer-critical claim is still scheduled. Unresolved support at expiry or stream end is flagged low-confidence.
Supervision starts from unimodal tool proposals structured by Gemini-3.6-Flash into a fact table. The forecaster is trained with cross-entropy then preference pairs; LoRA is SFT’d on observed and corrected states; on-policy updates reward accuracy, timing, and format. Paired CLEAN/MUTE/SWAP/MIX soundtracks with the same video teach the model to report sounds from audio, not from a visible source.
On SOVBench with audio-visual context, OST reaches 87.8 average accuracy against StreamOV’s 81.6; with extra QA context the scores are 88.4 vs 83.8. Recall moves from 73.2 to 82.5, and the answer/wait F1 from 90.5 to 92.4.
| Benchmark | OST | Baseline |
| SOVBench-O (AV) | 87.8 | StreamOV 81.6 |
| StreamingBench AV | 74.7 | StreamOV 73.1 |
| Video-Holmes | 60.0 | StreamOV 53.1 |
| Daily-Omni | 74.2 | StreamOV 69.3 |
| OmniVideoBench | 46.6 | offline Qwen3-Omni 38.40 |
On OST-DiagBench, d′ is 2.95 versus at most 1.38 for open models. CLEAN source recall is 95.49 with a MUTE hallucination rate of 10.39; naive-streaming Qwen3-Omni stays at 92.55 recall and 81.96 hallucinations. SWAP attribution is 81.76, MIX joint recall 51.37, and CLASH accuracy 46.47 against an open ceiling of 4.90.
Ablations cut the loop at each joint. Sharing the A/V budget drops Recall from 82.5 to 74.2. Retracting the claim without descendants drops it to 73.2. Verifying at issue time, before the window, drops the SOVBench average from 87.8 to 83.6.
Live assistants that watch screens and listen to speech fail in a specific way: they lock onto the first visual reading. OST is a state-machine patch, not a new 30B train. Freeze the backbone, keep claims pending, verify on the owning modality, and push the correction through memory.
The diagnostic is the part worth copying. Qwen3-Omni looks strong when audio and video agree, then hallucinates the visually suggested sound after the soundtrack is muted. Hearing a clip in isolation does not mean the joint model will use that audio.
There is no standalone limitations section. CLASH is still 46.47, so subtitle-speech conflict is far from solved. The StreamingBench lift over StreamOV is only 1.6 points; gains concentrate on recall and audio-using tasks. The stack is tied to Qwen3-Omni and a Gemini annotator. OST-DiagBench is author-built and screened for cross-modal competition. A frozen backbone also cannot repair encoder mistakes.