Four recent frames beat FluxMem, so MemInStream retests 3,799 memory questions

MemInStream: Memorizing or Merely Recognizing? Rethinking Streaming Video Memory Evaluation

Zhenqi He, Dunyuan Xu, Yang Bo, Kien T. Pham, Yanghao Wang, Shuhan Li, Xintong Guo, Yuyang He, Fang Liu, Huankang Guan, Jingyan Chen, Rui Liu, Long Chen

cs.CV, cs.AI

2026-11-11

MemInStream keeps a question only if evidence solves it and text or later video does not. 3,799 items across 17 tasks; StreamForest scores 37.06% on state tracking.

What problem this solves

Streaming models answer while the video is still arriving. Useful frames often sit tens of seconds to a few minutes before the question, and the raw video cannot be reopened later. Existing benchmarks still treat QA accuracy as a memory score. That number points two ways.

A miss is not automatically forgetting. Give recent video models the annotated evidence intervals directly, and they still fail nearly a quarter of the items. Those errors are perception or reasoning. A hit is not automatically retrieval either. On current benchmarks, SimpleStream, which sees only the latest four frames, scores above FluxMem, a method built to use historical memory. A text-only baseline is competitive on the same sets. Accuracy alone cannot show whether a memory module is doing any work.

Method

MemInStream stamps every question with a query time and one or more evidence intervals, all of which end before the query. Dialogue in contextual items is fed turn by turn, not pasted in as a full transcript when the question arrives. The model answers from whatever memory it keeps. It cannot reopen the source video or see future frames. Items are multi-answer: the number of correct options is hidden, and the predicted set must match the ground truth exactly.

Gemini-3.8-Flash and Doubao Seed-2.1-Pro vote at 4 fps. Both must agree.

The filter works like checking an exam before blaming memory: keep a question only if it is solvable with the source open and unsolvable once that source is shut. Humans then check the video, the timestamps, and whether the evidence is complete. Seeds come from StreamingBench, OVO-Bench, TeleEgo, PhoStream, and S-EMBER, plus newly retrieved clips and questions. Old items are not imported as-is.

Seventeen tasks fall into episodic retrieval, state tracking and updating, multi-evidence integration, and contextual memory. Labels follow the main memory demand, and categories need not be exclusive. The first two cover one-shot recall and states that change. The last two join separate moments, or bind a name or reference from dialogue onto later frames.

The set has 3,799 questions and 8,217 evidence intervals, 2.16 per question. Intervals average 9.98 seconds, median 6.0. Multiple intervals appear in 56.7% of questions. The median gap from the latest evidence to the query is 36 seconds, and from the earliest evidence start it is 96 seconds. For 34.6% of questions the latest evidence ends at least 60 seconds before the query. For 66.0%, the evidence starts at least 60 seconds earlier.

Results

Offline and proprietary models keep no visual memory across questions. They see 4 or 30 seconds before the query, sampled at 1 fps. Online models watch the stream in order and answer from what they retained. Training-free systems are FluxMem, Response-G1, and HERMES. Trained ones include StreamForest, AURA, JoyAI-VL-Interaction, and MOSS-VL-Realtime. The metric is exact-match accuracy. Category scores are macro-averages over tasks.

MethodMetricResult
StreamForestState tracking, macro avg37.06%
StreamForestMulti-evidence integration37.34%
Gemini-3.8-Flash, 30 framesMulti-hop binding13.07%
StreamForestMulti-hop binding23.86%
StreamForestEntity-identity binding / dialogue context52.54% / 18.28%
Gemini-3.8-FlashEpisodic retrieval, 4 to 30 frames26.38% to 44.96%
Doubao Seed-2.1-ProEpisodic retrieval, 4 to 30 frames20.93% to 38.63%
JoyAI, memory on vs offState tracking, up to 60 frames at 2 fps43.9% vs 28.6%

Among online models, StreamForest has the highest average across the four groups. Its state-tracking and multi-evidence scores also sit above the proprietary runs in the same table. Those runs have no persistent memory, only 4 or 30 recent frames. The gap is a gain over a truncated window, not a matched comparison of model quality.

Extra recent frames lift episodic retrieval and do not lift every other group. Moving FluxMem and Response-G1 from Qwen2.5-VL-7B to Qwen3-VL-8B raises all four categories. Per-task deltas are not reported. Near-frame accuracy moves more with whether the window contains the evidence than with a few extra frames. Under full overlap, Gemini-3.8-Flash scores 90.6% at 4 frames and 86.2% at 30. With no overlap those figures are 21.8% and 19.8%. Doubao Seed-2.1-Pro at 30 frames goes from 19.2% with no overlap to 76.3% with full overlap. Full overlap covers only 32 questions at 4 frames. At 30 frames the split is 1,650 with none, 820 partial, and 507 full.

AURA can use about the last 30 seconds of video. Accuracy is 36.6% when the latest evidence falls 10 to 30 seconds before the query, and 18.0% in the 30 to 60 second bin. From the earliest evidence, the same step drops 56.1% to 30.0%. At 300 seconds or more, StreamForest, Response-G1, and MOSS still score 44.1%, 32.2%, and 33.9%. JoyAI with memory on reaches 35.6% inside its limited recent window. Under a fixed budget of at most 60 frames at 2 fps, turning that memory on moves episodic retrieval from 31.8% to 41.0%, multi-evidence integration from 22.3% to 32.6%, and contextual memory only from 26.9% to 29.7%.

Why it matters

On older benchmarks, four recent frames can beat an explicit memory method, so a bump from history compression, a memory bank, or a scene graph is hard to interpret. This set allows the ablation that matters: freeze the recent-frame budget and toggle memory, or bin items by how far the evidence sits from the query. Once AURA's 30-second window is exceeded, its score falls from 36.6% to 18.0%. Methods that keep earlier evidence still land between 32% and 44% at 300 seconds.

The best online scores stall near 37%. The set is more useful for seeing which memory skill breaks than for ordering a leaderboard.

Limitations

The paper says the present duration does not cover truly long streams. Evidence and queries need a wider gap before long-horizon retention and updating are actually under test. The time bins are not a forgetting curve. Difficulty and mix change across bins, which the paper states outright.

The validators, Gemini-3.8-Flash and Doubao Seed-2.1-Pro, are also the proprietary systems being scored. A question remains only if both can answer it when the evidence is visible. Weaker open models can still miss because they cannot read the frames. The near-frame scores of the two validators are not an independent external test. The pool was cut on their mistakes.

Necessity does not drop questions whose recent-frame window happens to cover the evidence. The median gap to the latest evidence is 36 seconds, and 507 questions are fully covered at 30 frames. The main table is not recomputed after those items are removed. With no overlap, Gemini still scores 21.8% at 4 frames and 19.8% at 30. The paper assigns that residue to other visual cues, language priors, or chance, without a split. Human rejection rates and annotator agreement are unreported. Exact match on multi-answer items also lowers the headline numbers, so they should not be lined up against single-choice scores from older benchmarks.

Terms

Source

Related papers

All paper explainers