MobileMem: Learning from a Year of Mobile Experiences
Xinle Deng, Yida Xue, Xiangyuan Ru, Haoming Xu, Shuofei Qiao, Mengru Wang, Yijun Chen, Buqiang Xu, Chen Jiang, Yuchen Eleanor Jiang, Lizhong Wang, Jianfeng Wang, Li Zeng, Haofen Wang, Guilin Qi, Huajun Chen, Ningyu Zhang
cs.AI, cs.CL, cs.LG, cs.MA, cs.MM
2026-08-11
OPPO and ZJU-led collaborators release MobileMem, a year-scale on-device memory benchmark synthesized from real mobile usage traces; the best of nine memory systems (A-MEM) reaches only 79.7 overall, Mem0 lands at 35.6, and the top multimodal method scores 31.8.
Next-generation personal assistants need to remember months to a year of user experience, but existing memory benchmarks stitch together single sessions, lack temporal evolution, cross-app heterogeneity, and multimodality. Collecting real long-horizon personal data is expensive and privacy-bound. On-device adds its own constraints: tight storage and compute, memories fragmented across a dozen apps, preferences that drift, and the privacy requirement that processing stay local.
The core is KEME, a knowledge-guided synthesis engine that builds trajectories from structured prior knowledge instead of raw content capture. The text track interviews 2 volunteers for user profiles and collects only app-usage statistics and context signals (location, Bluetooth, network state) from their OPPO phones, never raw interaction content; LLMs then generate concrete interactions from app-specific message templates, segmented into sessions by timestamp. The multimodal track (MobileMem-Omni) adds 8 more volunteers plus 8 LLM-generated virtual users, a personal relationship graph, and portrait images, with Seedream as the primary generator and InsightFace verifying face consistency. Questions are synthesized bottom-up over a hierarchy: leaves produce single-point QA, internal nodes compose multi-hop, temporal, knowledge-update, and implicit-preference questions, with adversarial (unanswerable) questions as their own class.
On the text track, nine memory systems run with two LLM backbones, judged by Qwen3.5-397B. Overall scores under GPT-4.1-mini: A-MEM 79.7, HippoRAG2 78.9, Long Context (1M window) 54.5, NaiveRAG 37.2, Mem0 35.6, LangMem 24.8. A-MEM and HippoRAG2 lead because they preserve original conversations without aggressive compression or overwriting; Mem0's extraction prompt watches only user messages and misses facts from app and assistant turns. An adversarial-question inversion: LangMem, weakest overall, scores best there, because stronger retrievers surface weakly relevant but distracting memories that make the model confident an answer exists. A-MEM burns roughly 2.5M tokens per trajectory for memory construction, doubling under GPT-5.4-mini as keyword extraction explodes. On MobileMem-Omni (16 users, 19,060 images, 7,415 QA pairs, 1.72M context tokens per user), the best multimodal method M2A reaches 31.8 overall, multimodal long context 30.9, and text methods with GPT-5.1 image captions reach 25.9. A ceiling near one third means the field is wide open. Chinese questions score below English across methods, and adding captions to NaiveRAG helps visual questions while hurting text ones. Error analysis over 80 sampled failures: 13 of 20 Long Context failures are context-window truncation, 12 of 20 NaiveRAG failures are retrieval misses, and all four analyzed systems share answer-generation errors driven by distracting memories.
Agent-memory teams get a ready-made proving ground in both text and multimodal forms. The spread across nine systems is itself a selection table: routes that preserve raw conversations and boost recall via metadata and graphs clearly beat aggressive compression. KEME's synthesis-from-structured-priors idea transfers to any setting starved of long-horizon personal data. Adversarial questions and construction token cost are dimensions most benchmarks omit, and either can veto a deployment.
The authors list noisy synthesis, coarse user modeling, and evaluator coverage lagging the field. Reading closely adds more: the text track rests on just 2 real volunteer profiles, so user diversity is narrow, and 8 real plus 8 virtual in the multimodal track is only somewhat better. All trajectories come from GPT-5.2/5.1, so LLM distribution bias seeps in despite manual spot checks. LLM-judge reliability is never validated against human agreement. Only usage statistics are real; every interaction is synthesized, so the framing oversells how "real" the year of experience is.