StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding
Xichen Zhang, Guankai Li, Yinghao Zhu, Shijian Wang, Sitong Wu, Shaozuo Yu, Meng Chu, Yuan Lu, Jiaya Jia
cs.CV
2026-08-06
A benchmark of 243 hour-scale videos and 3,646 open-ended QAs; StreamMind's two-tier architecture beats streaming baselines on all four capabilities and cuts latency 66%.
Deploying multimodal agents in an always-on form (wearables, robots) means they must ingest continuous audio-visual streams, maintain hour-scale memory, and speak up on their own. But current evaluations mostly use short clips plus multiple-choice questions, and prior analysis showed a "last-four-frames" baseline can match or beat complex streaming models, because answer options leak language shortcuts. A high score does not establish real hour-scale understanding.
StreamArena is a benchmark for hour-scale, interactive, long-horizon streaming video understanding: 243 full-length videos (average 88.8 minutes) and 3,646 human-validated open-ended QA pairs (about 15 per video, draft retention around 73%), across four capabilities: real-time perception (RTP), historical retrospection (HR), proactive interaction (Pro.), and multimodal tool use (Tool). Open-ended answering removes the multiple-choice shortcuts. Historical retrospection carries most of the questions, with 1,732 tool-use and 774 proactive items, and the hardest questions span 15 evidence segments.
The companion StreamMind is a two-tier architecture: frontend workers handle latency-critical interaction and proactive monitoring on independent schedules, while backend workers asynchronously build persistent multimodal memory and do historical recall and external search. Router, Recall, Search, and Monitor workers split the work, moving perception and memory construction off the response-critical path so a query can reuse already-built state.
| Capability (shared Qwen3.5-397B-A17B) | Best streaming baseline | StreamMind | Relative gain |
| Real-time perception | -- | 44.5 | +58.4% |
| Tool use | -- | 56.1 | +228.1% |
| Proactive | -- | 11.6 | +54.7% |
Pooled query-to-answer latency drops from 81.4 s to 27.5 s (a 66.2% reduction) while retaining 89.7% of pooled accuracy. Human reference: RTP 91.8, Tool 95.2, Proactive 91.5.
One trade-off must be stated plainly: on raw accuracy, offline turn-based models (Gemini 3.5 Flash: RTP 51.3, HR 51.4) actually outscore StreamMind. StreamMind wins the "streaming-capable" category, trading absolute accuracy for continuous state and proactive support.
On the benchmark side, it puts streaming, long-horizon, proactive, and tool-use capabilities into one open-ended test instead of evaluating each in isolation. On the architecture side, StreamMind's bet is that persistent perception and memory construction must be moved off the answer-critical path, so a query just retrieves already-built state. The latency win comes from state reuse, not free computation.
Open-ended scoring relies on Gemini 3.1 Pro as a judge, importing LLM-judge bias. Proactive interaction is very hard overall: humans score 91.5% while StreamMind scores only 11.6%. The 243-video scale is moderate and coverage is limited. The fact that offline models score higher shows StreamMind's edge is coverage plus latency, not answering more accurately.