AskStream: proactive streaming models fail to ask, 7.22% Ask-F1 vs 52.18% for offline MLLMs

AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding

Lu Wang, Zhuoran Jin, Longxiang Wang, Junhong Chen, Yupu Hao, Yao Xu, Daojian Zeng, Yubo Chen, Kang Liu, Guang Liu, Jun Zhao

www

cs.CV, cs.CL

2026-10-29

AskStream adds an ask action to streaming video interaction. On 438 videos and 8 subtasks the best model reaches 52.18% Ask-F1; streaming proactive models manage 7.22%.

What problem this solves

Interaction paradigms for video understanding have moved from offline (watch everything, then answer) to streaming (answer as frames arrive) to proactive, where the model decides both how and when to answer, which matters for anomaly alerts, procedural guidance and continuous monitoring. Existing proactive benchmarks share a hidden assumption: the information a query needs will eventually arrive in the stream, so the only choices are wait or answer. In practice the gap never closes. A key entity is occluded, the viewpoint is wrong, the question is ambiguous, the user's preference is unknown. The model either waits forever or hallucinates. The authors ran a controlled check: with the key entity blurred, Qwen3.5 9B scores 42.3% on a five-way choice; after the model asks and receives the right evidence, accuracy climbs to 62.3%, nearly matching fully clear input at 63.1%. Irrelevant evidence barely moves it. Asking works, so the real question is whether models know when to ask and what to ask for. AskStream measures exactly that.

Method

Interaction is modeled as a POMDP; each second the model picks one of three actions: WAIT, ASK or ANSWER.

• Uncertainty has two sources. Video-side: explicit gaps (key entity blurred, missing viewpoint) and implicit gaps (personalized entity or action concepts the model cannot know). User-side: intent problems (ambiguous questions, false premises) and preference problems (unknown or outdated preferences). Four categories, eight subtasks.
• Two timing patterns. Some uncertainty can only be recognized once relevant footage arrives, so the model waits then asks; other uncertainty is visible from the start, so it asks first and waits for evidence. 438 videos (16.72 s to 118 min, 9 domains, 3.2 turns per video on average), 1,401 questions, 2,714 manually annotated frame-level windows; total wait/ask/answer window durations are 33.92, 30.47 and 12.00 hours.
• Ask-F1 combines two timing-aware scores: Sask for asking the right thing at the right time, and Sans for answering correctly and promptly once the information arrives, aggregated with a harmonic mean. Sans is computed only over turns with a successful ask, which separates failure to acquire information from failure to reason after acquiring it; otherwise one missed ask would be penalized twice. Earlier actions inside a window earn more credit, and swapping the linear decay for logarithmic or exponential weighting leaves the ranking of all seven models unchanged (Spearman ρ = 1.00).
• StreamHarness gives offline models hierarchical memory: the 10 most recent frames (1 fps) stay raw, every 10 frames compress into one mid-term summary, every four mid-term summaries merge into a long-term block, with at most eight blocks kept.
• Construction runs five stages: videos from YouTube, COIN, Ego-Exo4D, PEARL and Blender synthesis; manual annotation of queries and windows through a web interface; Grounded SAM 2 for entity segmentation and selective blurring; GPT-5.6-Sol to generate action and answer options; human verification against a blurred version and a revealed version of each video to confirm the former cannot be answered uniquely and the latter can.

Results

Multiple-choice setting, Ask-F1:

Key readings:

• Gemini's Sask (44.63) lags its Sans (62.80): asking is the weak stage. The other six streaming proactive models score zero on most subtasks.
• The subtask spread is wide: preference tasks are reachable (UPC 81.36%, UPM 64.03%) while viewpoint-missing (VIM 17.46%) and personalized-action (PACM 17.73%) collapse.
• Oracle ablation on KVEM: base 37.17%; telling the model when to ask lifts it to 58.30% (+21.13), telling it what to ask only reaches 40.96% (+3.79); both ask oracles give 81.0%, and all three decision oracles 83.3%. Early ASK is the dominant error, so timing is the bottleneck.
• StreamHarness matters: 52.18% versus 41.70% for a plain sliding window, 26.72% for uniform 128-frame sampling and 16.99% for 64 frames.
• Open-ended is harder: Gemini 29.19%, AURA 4.01%. Gemini-3.8 Flash serves as judge, with 93.50% agreement against human consensus on 300 sampled turns (human-human 96.17%) and no clear self-preference.
• Keeping dialogue history across turns barely helps video-side uncertainty but adds 4.40 points on AUQ and 5.38 on UPM; raising the ASK budget from 1 to 16 lifts scores monotonically before saturating around 52.35%.

Why it matters

This is the first benchmark to put an ask action into proactive streaming video, expanding the action space from two to three; the comparison table shows no prior streaming or proactive benchmark has an ask column. Teams building real-time visual assistants, remote guidance or monitoring can adopt the setup as is, and the diagnosis is sharp: the bottleneck is deciding whether and when to ask, not generating the answer, with the timing oracle alone worth 21 points. For training, that argues for frame-level timing supervision, and the 2,714 annotated windows are ready-made material.

To be clear about what this is: a benchmark paper. No new model, no training recipe; the contribution is the task definition, the metric and the annotation protocol.

Limitations

The authors list four: at most one ASK per turn, while real clarification can be iterative; no embodied agents that must interleave perception, asking and physical action; every interaction starts from an explicit user query rather than self-triggered awareness of insufficiency; the time windows are static, while real uncertainty evolves and overlaps.

Three more caveats. The 7.22% versus 52.18% gap conflates capability with protocol: offline models run through a purpose-built hierarchical-memory harness while streaming models use their own memory, so the two lanes are not strictly comparable, and attributing the entire gap to an inability to ask overreaches. The open-ended judge is also the top-ranked model, validated on only 300 turns. And the uncertainty is largely manufactured (Gaussian blur, withheld preferences); how far it transfers to natural occlusion and noise is untested.

Terms

• Proactive paradigm: the model decides when to respond instead of waiting for each user turn.
• POMDP: partially observable Markov decision process, a decision framework where the true state is hidden and decisions rely on partial observation.
• Ask-F1: harmonic mean of a timing-weighted asking score and a timing-weighted answering score.
• StreamHarness: the paper's hierarchical memory for offline models, raw recent frames plus progressively compressed summaries of older footage.
• Oracle ablation: feeding the model the correct decision at one stage and measuring how much score recovers, to locate the bottleneck.

Source

• arXiv 2610.38678v1
• PDF
• Hugging Face
• GitHub · https://github.com/wanglu-cs/AskStream

Related papers

• PhysInsert hits 63% text-guided DIoU while adapting pose and lighting
• IntReal: a 100K real-world dataset and 7-dimension benchmark for interleaved text-image generation
• One Shared Encoder, One Schema: Tencent's 5B Youtu-Parsing-Omni Sets OmniDocBench SOTA at 96.96
• Only low-frequency tokens go global: SpecSlot cuts pixel-space image generation cost 38%
• HairDeck: single image to rigged game hair cards, 33x faster than CHARM at 44% lower Chamfer error
• Storing VAE latents inside 3D Gaussians gets single-image world generation to 9 FPS

All paper explainers