Tencent & KCL release GroupAssistBench: group-chat agents recall memories but fail to use them
量子位 · wechat · 2026-11-21
Tencent Yuanbao and King's College London's NLP group released GroupAssistBench, built from 50 real group chats and 80,299 messages into 2,395 eval questions testing whether multi-user agents actually use memories in service. Paper, code and dataset are public.
- Two-level eval: L1 memory recall (1,374 fact questions) vs L2 open-ended service tasks (1,021, e.g. booking reminders that must follow the latest group decision), stratified by single-memory, multi-memory integration, and update/conflict dependencies.
- Key finding: EverMemOS beats full context on L1 recall (0.621 vs 0.570) but loses on L2 service (0.247 vs 0.431) — better recall doesn't mean better service. Among 497 paired memories recalled in L1 but not effectively used in L2, 36% never entered the model's input; 64% entered but weren't applied.
- Rubric scoring: L2 answers are judged with evidence-bound rubrics (grounding, service completion, penalties for stale info), with 89.3% agreement between auto-judge and human majority vote.
- Interventions: combining RER (relational evidence refinement) and ACB (binding facts into reply constraints with subject/scope/validity) lifts L2 scores of LightMem, xMemory and EverMemOS by 7.03, 4.64 and 4.43 points respectively.
More from coding & agent
- Dev likes Gemini 4 but finds Antigravity underwhelming; wary of account bans — jeremyphoward · 2026-11-21
- Rubrik eBook documents 15 real AI agent disasters with defenses — TheTuringPost · 2026-11-21
- Voice-to-app in minutes: Gemini Neon builds a working app with tests — LinusEkenstam · 2026-11-21
- mnemos: a self-curating agent memory plugin for Hermes that fades unused memories — RileyRalmuto · 2026-11-21
- Study: evolutionary LLM search found valid solutions but picked invalid programs in 2 of 3 runs — Unusual-Spot-5846 · 2026-11-21
- Dev routes tool calls through a 421M model at 12ms to catch risky agent actions — looselyhuman · 2026-11-21