Beyond Answering: Learning to Speak First and Act in Streaming Video
Shuhan Li, Huankang Guan, Xintong Guo, Yang Bo, Yuyang He, Dunyuan Xu, Jingyan Chen, Fang Liu, Tsz Kin Wan, Jinpeng Chen, Zhenqi He, Yuxuan Hu, Wenbo Li, Han Gao, Qijun Li, Long Chen, Hongsheng Li, Rui Liu
cs.CV, cs.AI
2026-11-13
AURA 2.0 (Huawei) adds model-initiated conversation, tool calling and visual grounding to streaming video LLMs, topping PhoStream at 65.69, 9.54 points above the best baseline.
Streaming video LLMs, which process video as it arrives and decide for themselves when to respond, have improved quickly, but the interaction protocol has not changed: the user asks a question or sets a monitoring goal, and the model decides whether to answer now, wait for more evidence, or answer in several passes. When nobody asks anything, the model stays silent, even when the scene clearly offers something worth saying. AURA 2.0, from Huawei Research with CUHK MMLab and HKUST, extends the question of when to speak into whether to open a topic at all: the model finds subjects worth discussing from the evolving visual stream and its own conversation history, then picks the moment to speak. In the paper's capability table, five comparable streaming models all lack this; by the authors' account, no existing model has been trained to start open-ended conversation without any prior request.
The architecture follows the previous AURA: video arrives in one-second chunks, and for each chunk the model emits either <|silent|> (keep observing) or <|respond|> (followed by an utterance, a tool call, or a boxed reply). All three new capabilities fit into that single format:
Training data extends AURA's set to 484,459 samples, 11.43B tokens, 5,400 hours of video, and 40M frames. Inherited QA tasks still dominate (Real-Time QA 34.2%, Proactive QA 22.1%, Multi-Response QA 12.4%), with MIC at 11.8%, IVG at 9.8%, and SATC at 5.3%. MIC openings are chosen by an MLLM; IVG data rewrites about 55,000 grounding and tracking trajectories from MeViS, Refer-YouTube-VOS, and similar sets; everything passes structural checks plus per-sample MLLM verification. Training is full-parameter SFT from Qwen3.5-9B with a frozen visual encoder, in three curriculum stages: streaming interaction plus tool use first, then MIC and IVG with 65.3% replay of earlier data to prevent forgetting, then mixed-capability trajectories, about 1,100 accelerator-hours in total.
The most reusable training detail is the handling of response-control imbalance: <|silent|> accounts for over 94% of supervised control decisions in every stage, with 24 to 42 silent steps per clip on average, so plain cross-entropy would train a model that never speaks. The fix applies a focal factor (γ=2) only to the two control tokens, averages each class separately at weight 0.25, and leaves all other tokens on standard cross-entropy at weight 1.0. No separate response classifier, no RL. Long streams are handled by LHIM: the active visual window is fixed at 30 seconds, video beyond it is sliced into 30-second segments that a separate summarization MLLM compresses into timestamped event-level QA pairs, and older records merge pairwise into coarser summaries at 5- and 10-minute thresholds, so visual context never grows with session length.
| Benchmark | Metric | AURA 2.0 | Best baseline |
| PhoStream | Overall | 65.69 | Doubao-Seed-1.8, 56.15 |
| StreamArena | Proactive | 12.8% | MOSS-VL-Realtime, 11.8% |
| StreamArena | Tool use | 18.5% | MiniCPM-o 4.5, 17.1% |
| ProactiveVideoQA | WEB (ω=0.5) | 58.8 | MOSS-VL-Realtime, 55.4 |
| IPIBench | Proactive task mgmt avg | 24.99% | highest, incl. proprietary |
On PhoStream the Overall score beats Doubao-Seed-1.8 by 9.54 points, Forward leads MOSS-VL-Realtime by 37.95, and the early-response rate is 1.30% against 41.51% for MOSS and 94.93% for JoyAI, which says the timing control actually transferred. The cost is a 23.9% no-response rate, so missed replies remain common. On IPIBench, Timing is 49.30% against GPT-4o's 50.59%, and the task-management average tops every closed model. ProactiveVideoQA leads on three of four categories but trails MOSS on TV (49.3 vs 50.2).
Targeted evaluations of the three new capabilities all run on small internal sets. MIC-Bench (25 videos, judged by gpt-4o): with dialogue context, trigger rate is 80% (20/25) at 4.10/5 quality; visual-only trigger rate rises to 96% but quality falls to 3.62/5, with 93 Good and 15 Bad out of 173 messages, at 3.0 to 3.5 messages per minute. SATC on 167 questions: enabling tools lifts accuracy from 35.9% to 48.5% (+12.6 points), with calls on 41.3% of questions. IVG on 50 COCO images: mIoU 0.906 versus 0.635 for the Qwen3.5-9B backbone, with the gain almost entirely from fewer missing boxes (37/37 vs 26/37), since IoU on the 26 queries both answer is level (0.906 vs 0.903); correct abstention on absent targets regresses (11/13 vs 13/13).
Serving: single-session time to first token is 157 ms and time to first audio packet 344 ms; at 16 concurrent sessions, 234 ms and 601 ms with RTF 0.203. Three optimizations cut median TTFP from 758 ms to 344 ms (54.6%), with per-session frame caching contributing about 86% of the gain.
Proactive interaction is where streaming video actually ships: in glasses, dashcams, and monitoring, users will not keep asking questions, so the assistant has to judge for itself when speaking up is worth it. Existing open-source proactivity answers standing instructions set by the user; AURA 2.0 turns request-free open-ended initiation into a trained capability and releases code and model. For teams building streaming systems, three pieces port directly: the focal-loss recipe for imbalanced response-control tokens, the LHIM-style tiered memory, and the serving-level latency work. The benchmark results put it at the top of the open-source streaming field, but the margin over JoyAI and MOSS is mostly single-digit points: steady progress, not a generational gap.
The authors list two: MIC usefulness is inherently subjective and depends on user preferences for topic, timing, and frequency, which the system does not yet model; and the cascaded ASR-VideoLLM-TTS pipeline cannot listen while speaking, with full duplex left to future work. A few more discounts apply. MIC-Bench is an internal, unpublished set of 25 videos with quality judged entirely by gpt-4o. Visual-only quality drops to 3.62/5, so topic selection clearly degrades without a user interest to anchor it. MIC training timings are MLLM-synthesized and filtered, with no validation against real user preferences. IVG abstains on absent targets less reliably than its own backbone, and the PhoStream no-response rate sits near one in four. The public evidence for the headline capability is four benchmarks; conversation initiation itself rests on internal evaluation only.