OneStreamer: A 4B streaming video LLM that writes its own caption memory, beating 11B models on all 8 streaming benchmarks

OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction

Xiangyu Zeng, Yuandong Yang, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Qingyi Si, Dingyu Yao, Changlian Ma, Haoran Chen, Xinyu Chen, Yansong Shi, Junhao Zhou, Yifei Li, Jun Zhang, Chuanyu Qin, Chenxu Yang, Xinlei Yu, Kun Ouyang, Yuchen Shao, Qianshan Wei, Changhai Zhou, Jun Gao, Jiaqi Wang, Limin Wang

cs.CV

2026-10-01

OneStreamer, a 4B streaming video LLM from NJU and collaborators, learns to write hierarchical caption memory while watching and to respond only when evidence suffices, ranking first on all eight streaming video benchmarks it evaluates, ahead of the 11B MOSS-VL-Realtime.

What problem this solves

A model watching a live video stream faces two problems at once. Evidence retention: a scene that looks irrelevant now may be asked about minutes later, by which point its source frames have slid out of the limited visual context. Response timing: an answer must land when sufficient evidence exists, not before (language-prior guessing) and not after (the moment has passed).

Existing routes each miss something. Memory mechanisms that compress or retrieve historical visual tokens compete with recent frames for context capacity and weaken real-time perception. Streaming-thinking methods maintain evolving reasoning, but a reasoning trace is not a time-grounded, reusable factual record. State-token response control suffers label imbalance: silence dominates, and dense supervision teaches the model to keep waiting. OneStreamer's bet is to put perception, memory writing, and response triggering into a single causal generation process with one shared set of control tokens.

Method

Initialized from Qwen3-VL-4B-Instruct with a frozen vision encoder, the model keeps a Recent-N FIFO sliding window of visual tokens. Two mechanisms carry the design.

Proactive Hierarchical Caption Memory (PHCM) handles retention. While observing, the model generates two kinds of time-aligned text records: </Observe> introduces dense local-detail captions over short intervals (objects, actions, scenes, state changes), and </Summary> introduces sparser summaries of completed events. Each record is appended to the text history and stays available after its source frames leave the visual window, so distant events live in compact text without eating into current visual capacity. Training uses causal streaming caption targets, so every record describes only what has already been observed.

Proactive State Transition Learning (PSTL) handles timing. Control states are </Silence> (keep observing), </Standby> (evidence emerging but insufficient), and </Response>. Under dense supervision, silence tokens vastly outnumber output tokens. PSTL groups state tokens by their transition direction (previous state to current state), sets a supervision quota equal to the largest group targeting an output anchor, and uniformly subsamples every group down to that quota. All output anchors keep full supervision while representative state-change and state-persistence tokens are retained. Unselected tokens are masked only from the state loss; they stay in the causal sequence, and caption and task-text supervision is untouched.

A data synthesis pipeline supports both: the caption branch converts verified multi-granularity annotations into evidence-aligned streaming targets, while the QA branch localizes a coarse evidence interval and then calibrates the response time to the earliest reliable moment, advancing annotations by 13.72 seconds on average. Combined with cleaned open-source data, this yields OneStreamer-1M, 1.16M records decontaminated against every benchmark's test split at the source-video level.

Results

First place on all eight streaming benchmarks, against specialized streaming models from 4B to 11B.

That is a 25.0% average relative gain over the Qwen3-VL baseline and 6.1% over the strongest competitor per benchmark. It leads the 11B MOSS-VL-Realtime by 1.9, 4.0, 13.1, and 7.4 points on the four perception-and-memory benchmarks.

The ablations carry the argument. With the same Recent-16 window, retaining generated captions lifts OVOBench-Backward ASI from 63.5 to 71.6; against the full visual history variant, PHCM scores 4.0 points higher on ASI (71.6 vs 67.6) while matching real-time perception (81.4 vs 80.9). It exceeds both single-sided baselines rather than trading one off. On state supervision, PSTL trains on 27.5% of state tokens and beats dense supervision with focal loss on all three proactive benchmarks (48.7/36.6/41.6 vs 47.0/32.8/26.5); a random-sparse baseline at the same ratio collapses to 26.6/27.4/1.4, so the gain comes from the selection rule, not sparsity. On a 360-second sample, PHCM cuts context from 62,094 tokens to 4,308 (93.1% less) and TTFT from 4.560s to 0.124s, adding only 1,272 tokens and 0.030s over a plain FIFO window.

Why it matters

This moves streaming video memory from an architecture problem to a generation problem. No retriever, no memory encoder, no revisiting discarded visual features: memory is just text the model wrote itself, timestamped, sharing one generation interface with the answer. For live-stream assistants, wearable agents, and monitoring systems, the engineering takeaway is concrete. A recent visual window plus caption memory gets close to full-history recall for about a thousand extra tokens and 30 milliseconds of latency.

The data pipeline is reusable on its own. Separating evidence localization from response-time calibration, and advancing response annotations by 13.72 seconds on average, is a direct reference for anyone building proactive-response datasets. Code and data are open (github.com/MCG-NJU/OneStreamer), and OneStreamer-1M, decontaminated at the source-video level, is among the broadest streaming-interaction training sets available.

Limitations

Both the paper's own admissions and what reads as weak spots:

• Memory is text, so fidelity is bounded by caption quality. Fine-grained appearance (color, faces, spatial layout) necessarily degrades through transcription; Full still edges PHCM on OVOBench-Backward EPM (63.0 vs 62.6), so visual detail is not fully captured.
• Response-time calibration relies on Qwen3.5-series scorers, and the authors' own manual inspection concedes the calibrated time is not always the earliest answerable moment; it is a trade-off between timeliness and evidence sufficiency.
• Among the compared methods, several models without officially reported numbers were re-evaluated by the authors, with the higher of two settings reported at the whole-benchmark level. That favors the comparators in intent, but benchmark-level selection still leaves room for interpretation.
• Training used 32 H200 GPUs, which is not light for most teams, though the 4B scale and single-stage SFT recipe are restrained.
• All evidence comes from academic benchmarks. Real deployments bring latency jitter, missing audio, and multi-turn user interruption, none of which are tested here.

Terms

• Streaming video LLM: a model that ingests video incrementally with no preset length, seeing only content up to the current moment.
• FIFO sliding window: keeping only the most recent N frames of visual tokens, first in first out; cheap, but discards distant history.
• Control token: a special marker in the generated sequence that declares a state (observe, standby, respond) instead of producing text.
• Output anchor: a control state that initiates text output, such as </Response> or </Observe>; PSTL keeps supervision on all of them.
• TTFT: Time To First Token, the latency before the first token appears; in streaming settings it decides whether a response is timely.
• Decontamination: verifying no training sample shares a source video with a benchmark test split, including non-overlapping clips of the same video.

Source

• arXiv 2610.01762v1
• PDF
• Hugging Face
• GitHub · https://github.com/MCG-NJU/OneStreamer

Related papers

• Column shuffle self-pairs lift variable-length scene edits to 47.6% accuracy
• Same-time reallocation lifts HunyuanVideo sparse attention by 11.8 dB PSNR
• Frozen ALPS tokens hit 85.5% on LIBERO while matching V-JEPA 2-L
• Training-free segmentation: a frozen MLLM draws B-spline masks itself, 71.5 cIoU on ReasonSeg
• Wrong tap, same login screen: LGWM hits 0.987 AUC on GUI transitions in 17 ms
• Asymmetric distillation lifts 10-turn consistency from 3.03 to 3.85

All paper explainers