MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories
Guangzhi Xiong, Xinyuan Zhang, Xiao Yang, Hyokun Yun, Kai Zhang, Shiun-Zu Kuo, Hyeonjeong Ha, Xilun Chen, Kai Sun, Lucas Liang, Guangqiang Dong, Ejaz Ahmed, Ahmed A Aly, Anuj Kumar, Raffay Hamid, Aidong Zhang, Xin Luna Dong
cs.CV, cs.AI, cs.CL
2026-10-01
MemLife writes egocentric video into first-person text memory read by a time-indexed agent, beating training-free baselines by 4.6–12.0 points; MemOpt's RL writer adds 2.7–5.0.
Smart glasses and body-mounted cameras can record months or years of egocentric video. Answering "where did I leave my ring last month" looks like retrieval-augmented generation, but reprocessing raw clips for every query stops scaling once histories reach hundreds of hours: latency budgets are tight, context windows hold few frames, and daily scenes repeat with subtle differences.
The alternative is to compress video offline into text memory and query only the text. Text is orders of magnitude more compact than frames, natively consumable by LLMs, and readable by users. Existing systems fail in two characteristic ways: the writer drops evidence a future question needs, or the retriever cannot locate the right entry as the search space grows and similar entries compete. A third failure is quieter: writers hallucinate content the video never showed, planting false evidence. Compression itself is a tradeoff between discarding information future questions ask about and retaining trivia that dilutes retrieval.
The work proceeds in two steps: design a strong writer by hand, then learn a better one.
MemLife's writer turns each segment into one text entry, independently, with three deliberate choices. Multimodal fusion: frames and speech transcripts are read together, because evidence for a future question may live in either modality. Entity grounding: names spoken in the audio are aligned to the people and objects on screen, so entries say "Jake is setting the table" instead of "a man nearby," preserving a cue retrieval can match. First-person narration: users ask about their own lives as "I", so the writer narrates as "I" too, narrowing the query-memory wording gap. Per-segment independence keeps compute and storage linear in history length with no error propagation; conditioning on preceding context did not help (Appendix I). The reader is a multi-turn agent with five actions: rewrite the question into a search query or time interval, similarity-search memory within an interval, fetch all entries in an interval, fetch raw video fragments (disabled when source video is absent), and answer. Time anchoring scopes searches to intervals like "last month," and retrieved entries are presented in chronological order, which for a question like "did I add milk before eggs" is the answer itself.
MemOpt is the more consequential half. Prior RL for video QA rewards final answers, but answer quality mixes in the execution noise of the reader's tool calls, making supervision noisy and expensive. MemOpt freezes the reader and trains only the writer against FIRM, a reward on the memory itself. Faithful is scored per token: a frozen evaluator reproduces supported content and minimally corrects unsupported spans, so hallucinations are localized to specific words. Informative checks whether the entry entails the source-grounded key fact behind each verified question-answer pair; questions are used only to build the reward, keeping the writer question-independent. Retrievable replays the reader's cached memory-access actions to check whether a candidate entry actually comes back, without rerunning reader reasoning. The three rewards multiply, advantages are computed per token after averaging within each candidate so long memories do not dominate normalization, and training follows GRPO. An information-theoretic identity backs the design: the extra uncertainty from using retrieved context instead of video splits exactly into a writing loss plus a recall loss, matching the informativeness and retrievability rewards, while faithfulness separately keeps memory grounded in the video.
Writer and reader both default to Qwen3.5-9B. MemOpt trains only on SuperMemory-VQA subjects S1-S6 and tests on S9-S10; EgoLifeQA, SuperMemory-LVQA (all ten histories merged, with cross-subject distractors), and EgoLife-EQA (recurring-event questions) are test-only.
| Method | SuperMemory-VQA | EgoLifeQA | SuperMemory-LVQA |
| EgoRAG (best training-free) | 49.28 | 48.20 | 47.51 |
| M3-Agent (best prior trained) | 53.93 | 35.40 | 54.90 |
| MemLife | 56.50 | 52.80 | 56.18 |
| MemLife + MemOpt | 60.35 | 56.60 | 58.91 |
| Oracle (all annotated evidence) | 67.58 | 66.20 | 67.58 |
Training-free, MemLife beats the strongest training-free baseline by 4.6-12.0 points across four benchmarks; with MemOpt the combined system exceeds the strongest prior baseline by 4.0-17.0 points while shrinking memory size by up to 31x (Appendix E). Enabling video access (MemLife-V) changes results only modestly, so the gains do not come from revisiting recordings. Training transfers: a 27B writer, a 27B reader, or plugging MemOpt into EgoRAG all help, though the 27B reader lifts absolute accuracy to 68.06 and shrinks MemOpt's margin, showing reader capacity compensates for some memory deficiencies. Supervision ablations: teacher distillation from a 27B model reaches only 57.78, end-to-end answer reward 57.46 with inconsistent transfer; retrievability alone lifts recall on both benchmarks but not accuracy reliably (53.45 on SuperMemory-VQA); adding informativeness reaches 60.03 but does not transfer to EgoLifeQA; full FIRM tops both at 60.35. A side observation: several trained baselines (VST 33.60, M3-Agent 35.40 on EgoLifeQA) underperform the training-free EgoRAG, consistent with end-to-end task rewards being noisy supervision. Table 2 makes the corrections concrete: the zero-shot writer mistakes lentils for corn kernels while MemOpt fixes the object without losing the surrounding action, and replaces a vague pronoun with the named person and the baked chicken the answer needed.
For memory-system builders, MemLife's three writer rules plus time-anchored retrieval are a copy-ready checklist worth 4.6-12 points with no training. For RL practitioners, MemOpt validates an unusual target: rewarding an intermediate representation rather than the final answer gives cleaner and cheaper supervision because reader execution noise is removed from the signal. The 31x memory reduction is a deployment constraint, not a bonus; hundreds of hours of video will never fit on glasses. The honest framing: most single-benchmark gains sit in the 4-6 point range, solid incremental progress whose weight lies in the paradigm and its consistency across settings.
The paper has no standalone limitations section; the one acknowledged in the analysis is that stronger readers erode MemOpt's gains. Reading closely, more concerns stand out. The oracle gap remains over 7 points (60.35 vs 67.58 on SuperMemory-VQA) and EgoLifeQA recall is 50.60, so information loss through text memory is real. The informativeness reward requires verified question-answer pairs per segment, which domains without dense QA annotation cannot provide. The faithfulness evaluator and the entailment judge are frozen copies of the writer model itself, judging its own output without external validation. The retrievability reward replays one default reader's cached actions, biasing writing toward that reader's behavior; transfer is demonstrated, the bias is not removed. And the accuracy-recall inconsistency on SuperMemory-LVQA is explained post hoc as cross-subject interactions providing useful unannotated context, plausible but unverified.