EM^2Mem: Event-Centric Multimodal Memory for Large Language Models
Yijun Chen, Yaqi Zheng, Yanya Li, Boyi Xiao, Buqiang Xu, Shuofei Qiao, Jizhan Fang, Xinle Deng, Yunzhi Yao, Xuehai Wang, Liuxin Zhang, Hui Li, Huajun Chen, Shumin Deng
EMNLP 2026 findings
cs.CL, cs.AI, cs.LG, cs.MM
2026-09-01
ZJU and Lenovo bind long-video evidence to event cells at construction time. Accuracy rises 2.0/2.4/3.7 points vs the strongest memory baseline, with 63.66% fewer inference tokens.
Long-video QA scatters its evidence. Frames, speech, OCR, objects and habits may sit minutes apart. Most multimodal memories and video RAG stores cut by modality: captions, keyframes, summaries, triples, graphs. Searchable, but the hits are fragments. The language model then has to rebuild cross-modal and temporal alignment at generation time, when the context budget is already tight and attribution is hardest.
This paper changes the retrieval unit from fragments to events, drawing on event cognition: people organize perception, language and action around event boundaries. Questions also tend to ask about events, situations and recurring behaviour, not isolated frames or graph edges.
EM2Mem is align-then-retrieve. The video is cut into roughly 30-second base segments. Each segment gets an event anchor, a temporal address, not content. Under that anchor sits a multimodal record: keyframe caption, dialogue, representative frames, structured fields for action, objects, topic, scene and entities. Multi-scale temporal views, 3 minutes, 10 minutes, 1 hour, hang off every cell they cover.
Two light graphs span events. An episodic graph links people, objects, places, topics and adjacent transitions. A semantic graph stores habits, preferences and stable relations, each pointed back at supporting anchors. At inference the system selects cells from local records, multi-scale context and graph cues, expands lightly, lets an LLM selector filter, and compiles a query-specific evidence view. Keyframes are a post-retrieval check, not the primary store.
Baselines cover base MLLMs, long-video LLMs, RAG and memory systems. The closest memory baseline is WorldMM, reported both as published numbers and as WorldMM† under the same evaluation. Ego-R1 Bench uses the published WorldMM score only; reproducing it would mean rebuilding subject-level memories for six participants.
Average accuracy beats the strongest memory comparison by 2.0, 2.4 and 3.7 points on the three benches.
| Bench | EM2Mem | Comparison | Delta |
| EgoLifeQA | 66.0 | WorldMM† 64.0 | +2.0 |
| Ego-R1 Bench | 67.7 | WorldMM 65.3 | +2.4 |
| Video-MME (L) | 76.8 | WorldMM† 73.1 | +3.7 |
Against the original WorldMM numbers the averages still edge ahead, though WorldMM remains stronger on several habit, temporal and synthetic categories. On EgoLifeQA, dropping temporal views falls to 60.4 (-5.6), semantic memory to 61.4, the episodic graph to 61.6. Local 30-second records alone still score 64.0, so short factual questions already have enough in the cell; relation and task questions are what need cross-event aggregation.
Efficiency on EgoLifeQA: per-query latency 459.00 s to 98.21 s (4.67×); wall-clock 229,502 s to 6,138 s; total inference tokens 42.03M to 15.27M, down 63.66%. WorldMM still builds a HippoRAG graph at inference; EM2Mem reads prebuilt event cells and runs 8 workers. Offline construction is heavier; wall-clock breaks even around 23 to 24 queries.
Strict 30-second event-level Top-5 recall is 30.8, 7.0 points above WorldMM after five iterative rounds; Top-1 is 23.0, close to WorldMM 5R at 23.8. On the first 250 EgoLifeQA questions, structured event fields with construction-time unification reach 71.2, above raw frames and flattened captions. Three keyframes take accuracy from 63.2 (none) to 66.0.
For long-video agent memory this is a change of retrieval grain: align at construction, read out whole events, do not stitch at generation time. Accuracy is a modest two to four points. The harder numbers are latency and tokens. The design fits videos processed once and queried many times, not a live stream. Code is slated to land in LightMem.
Structured fields buy searchability and drop fine pixel detail: small objects, colour, layout, subtle visual state, which the authors flag. The answer stage still uses selected keyframes, so some cross-modal alignment is deferred. Captioning and object-extraction errors from upstream MLLMs write into cells and graphs. Construction cost amortizes only after a couple of dozen queries, longer for tokens. Ego-R1 compares against published WorldMM, not a reproduction. Event cuts are a fixed 30 s clock, not necessarily true event boundaries. Habit questions benefit less from keyframes, so long-term patterns still live mainly in the semantic graph.