APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants
Jianguo Huang, Jinming Liu, Qiyao Wang, Liang Xu, Jianhang Li, Zhimian Wen, Mingda Li, Shule Lu, Zhicheng Wang, Yuhan Guo, Xin Jin, Wenjun Zeng
cs.CV
2026-09-29
APM-Bench tests cross-session persistent memory on 104 egocentric trajectories: raw-video replay is strongest but costs 3 GiB/hour; no system balances utility, latency and storage.
Most research on memory in streaming video models stays inside one continuous video: an event at minute two, a question at minute ten. Real assistant use is intermittent. Users take off smart glasses and put them back on hours or days later; the stream in between is never replayed, so everything the assistant knows about earlier sessions must live in a persistent, storable memory. Existing benchmarks either test a single video (StreamingBench, OVO-Bench) or stretch interaction to hour scale (StreamArena), and none scores memory utility together with its storage and latency cost. APM-Bench fills that gap.
The authors reorganize two egocentric datasets, EgoLife and HD-EPIC, into activity-linked multi-session trajectories: 104 trajectories, 549 sessions, 2,719 questions. A trajectory averages 69 minutes of video but 58.9 hours of wall-clock span, deliberately separating video duration from elapsed time, which also makes storage and latency comparable per video hour. Construction ran in two stages. GPT-5.4 mined 165 trajectory proposals from EgoLife daily summaries and humans kept 76; HD-EPIC recipe annotations contributed 28 more. Gemini-3.5-Flash captioned 30-second clips, DeepSeek-V4-Flash and GPT-4o proposed 6,412 candidates, and four filters (shortcut questions answerable without video, causal-timestamp violations, a VLM-agent audit, human review) left 2,719. Two annotators agreed on 300 sampled questions with Cohen's kappa 0.868.
Twelve tasks cover three capabilities:
Open-ended answers get a gated LLM-judge score: a wrong intervene/silent decision scores zero; passing responses are rated 1-5 by DeepSeek-V4-Flash against a reference. A separate 260-question set gives models only the two most recent completed sessions; half the questions have evidence inside that window, half outside, with a fifth option to declare insufficient evidence.
| Memory setup | Cross-session Und. | Real-time Perc. | Adaptive Resp. | Overall | Storage/hour |
| Recent 4 frames (no memory) | 27.21 | 53.06 | 32.48 | 37.58 | – |
| Gemini 3.6 Flash, raw-video replay | 69.37 | 64.20 | 47.05 | 60.21 | 3.01 GiB |
| Gemini 3.6 Flash, text summary | 47.50 | 65.98 | 47.14 | 53.54 | 3 KiB |
| Seed-2.0-Lite, raw-video replay | 54.47 | 50.75 | 41.88 | 49.03 | 3.01 GiB |
| OASIS, event tree (best specialized) | 35.46 | 52.92 | 35.99 | 41.46 | 2.66 GiB |
| HERMES, KV cache | 34.12 | 42.04 | 23.05 | 33.07 | 0.67 GiB |
The simplest strategy wins on recall: storing prior-session raw video and replaying it at query time reaches 69.37 on cross-session understanding, 21.87 points above text summaries, at GiB-scale storage and 34.11 seconds of end-to-end latency for Gemini. Text summaries cut storage to kilobytes and latency to 11.25 seconds, cost about 22 points of cross-session recall, and improve real-time perception for most models (Gemini 64.20 to 65.98, Seed 50.75 to 52.44): loading irrelevant history interferes with understanding the current scene. Among eight specialized systems, event-structured OASIS leads overall and is the only one supporting both streaming input and persistence; the KV-cache method ReKV needs 20.84 GiB per hour, more than storing the raw video.
Adaptive Response is the hardest family. With an explicit registered trigger, Gemini scores 73.62 (RCR) and 61.80 (PRM); on autonomous MPA and TPG it drops to 26.52 and 25.58. An oracle that feeds in exactly the needed evidence still leaves adaptive scores at 28.98-51.86, so the bottleneck is connecting past evidence to the current scene, not retrieval. The evidence-availability set adds another signal: Gemini answers 85.38% correctly when evidence is present but admits missing evidence only 59.23% of the time, while the four-frame baseline shows the mirror failure (3.85 vs 95.38). Recall and awareness of what you no longer have are separate skills.
Teams building glasses-style assistants get an open-sourced testbed that puts what-to-store, when-to-use, and how-much-it-costs on one scoreboard across 12 tasks. For method work the signals are clear: raw-video replay remains the strongest baseline and should be the comparison target for any memory-compression paper; event-structured memory is the most promising compression direction; autonomous intervention is the open gap.
The paper states plainly that no existing method achieves high utility, low latency, and low storage at once; this is a measurement study without a proposed fix. Data comes only from EgoLife and HD-EPIC, skewed toward kitchen and household activities, with no validation on other domains. Three fairness caveats: storage for specialized systems is estimated from inference-time state, though most were never designed to export persistent state; open-ended scoring relies on a single judge with no cross-judge robustness check; StreamForest and VST produce 36.14% and 37.05% unrecoverable format failures, so their low scores mix weak capability with poor instruction following.