GROVE: Growing and Reasoning over Temporally Stratified Memory from Streaming Video Experience
Sitong Gong, Caixin Kang, Tianyu Yan, Guo Chen, Bo Zheng, Kaipeng Zhang, Yunzhi Zhuge, Xiang Ruan, Huchuan Lu, Yifei Huang
cs.CV, cs.AI
2026-08-03
GROVE grows a stratified memory from a video stream without training, with one memory serving both reactive QA and proactive assistance, leading several long-video benchmarks.
A wearable assistant (smart-glasses type) needs to answer "when did I last see this person" and also proactively remind "you should take your meds." Existing systems mostly do only the first, reactively. Doing the second has meant bolting on a separate memory and control mechanism, leaving two disconnected systems. GROVE aims to do both from one memory grown causally from the video stream, retrievable when a question asks and activatable when the current situation calls for it.
Memory has four strata, grown online without retrospective passes: at the base, a perceptual trace (object counts and attributes, subject-action-object events, on-screen text); above it, moments with timestamps, roles, and salience; then episodes, variable-length activity segments with summaries and spans; at the top, patterns of recurring cross-episode behavior with frequency and typical times. Each window gets dual perception (a narrative caption plus a structured registry), and an LLM judges activity continuation to segment episodes. When an episode closes, it is atomized into moments and batch-matched against existing patterns.
Retrieval has four scale-native skills: perception lookup (exact counts, attributes, text), moment recall (what-when facts by keyword or time), episode replay (activity segments), and pattern traversal (cross-period queries). Reactive QA and proactive assistance share this memory and skill set: for QA the agent runs a bounded loop of at most 8 rounds picking skills by cue; for proactive service it fills evidence slots from the current perceptual state via a fixed schema. The whole system is training-free, using only frozen models (Qwen3.5-35B-A3B for perception, GPT-4.1-mini for consolidation, GPT-5.2 and GPT-5-mini for reasoning).
On the proactive-service benchmark EgoServe, GROVE scores Macro-F1 12.6, well above EgoMemo (8.0), GPT-5-mini (4.7), and Qwen3-VL-Plus (3.5), ranking first on 7 of 10 service subtypes with the largest gains in safety, tool use, and error recovery. On the long-video QA set MM-Lifelong (181 hours), it scores 23.50, 22.75, and 19.98 on day, week, and month splits, beating ReMA and GPT-5. Streaming understanding leads on OVO-Bench (67.5 overall), StreamingBench (65.1), and ESTP-Bench (28.6).
Ablations show the hierarchy earns its keep: removing the episode layer hits day-scale hardest (18.75 down to 14.57); removing patterns cuts the week split sharply (19.75 down to 13.00); disabling retrieval collapses accuracy (day 18.75 down to 7.25). With the full hierarchy, the agent needs fewer retrieval rounds (6.25 down to 4.95) and lower latency (59.3s down to 47.6s).
For smart-glasses and embodied-assistant teams, GROVE offers a deployable, training-free long-video memory that unifies reactive QA and proactive prompting in one structure. Stratification lets retrieval operate at the right scale, saving tokens and latency, which matters on edge devices. It also models a construction direction: use large models as ready components and put engineering weight on memory structure and retrieval policy instead of end-to-end retraining.
The authors list several: it inherits errors from the frozen perception model; evidence missed at ingestion cannot be recovered; higher strata update only when an episode closes, so live queries rely on the perceptual trace; cost grows with history; and different benchmarks used different backbones, so absolute cross-benchmark comparison is shaky.
One more from reading it: the system offloads reasoning to GPT-5.2 and GPT-5-mini, and how much of the numbers come from the memory structure versus a stronger reasoning model is not fully separated by the ablations. Training-free also means re-tuning prompts for every new scenario.