One Assistant, Many Memories: Benchmarking Real-World Group Assistance When Recall Is Not Enough
Zhanghao Hu, Linhai Zhang, Qian Zhao, Qi Zhu, Yuan Hua, Di Liang, Jiasheng Si, Xin Zhao, Yulan He, Zhumin Chen, Lin Gui
cs.CL, cs.AI
2026-11-03
On 80,299 messages from 50 real groups, EverMemOS beats full-context recall (0.621 vs 0.570) but scores 0.247 vs 0.431 when the reply must help the group.
A group assistant gets one reply. The facts it needs were left by different people on different days. In one trip chat the departure moves from May 1 to April 30, someone wants a window seat, and a no-show owes the group 500 yuan. The old date is still in the log. A reminder that says May 1 is a failed service.
Group-memory benchmarks such as GroupMemBench, EverMemBench, and GateMem use synthetic chats and score fixed answers, multiple choice, or a closed set of actions. They can check which value is current. They do not check whether a system that found the right memory can handle an open request. The split already shows up for a single user: on AlpsBench, memory lifts persona fit from 0.41 to 0.72 and drops emotional appropriateness below the same backbone with no memory. A group cannot peel the reply apart per person.
GroupAssistBench is built from deployed Chinese groups in Tencent Yuanbao, with King's College London on the team. It holds 50 groups, 80,299 messages, and 2,395 questions sampled from about 9,500 validated candidates: 1,374 recall items (L1) and 1,021 assistance items (L2). The chats cover trips, classes, hobbies, and project coordination. Construction cost more than $53K. Text is pseudonymized and rewritten. Raw logs stay private.
L1 asks about the history. Atom Recall is the share of required atomic facts a reply recovers. L2 asks for a plan, a recommendation, or a reminder, and there is no single correct answer. Each L2 rubric has three parts. Anchored facts must show up in the reply, with weights from 1 to 3. Failures distilled from real user complaints are penalized, with weights from -1 to -3: distorting a fact, undoing a settled plan, breaking an agreement or a member's constraint, misreading someone's current state, and repeating a value that was already replaced. A third block scores the request itself.
M1 depends on one memory. M2 combines several, and 56% of its anchors come from at least two members. M3 adds a conflict: the new value has replaced the old one, the question never says so, and 93% of its anchors also span members. A request is kept only when the same model, run with and without the history, gives a clearly better, group-specific answer once the history is present. Every memory is tied to a verbatim span. Conflict pairs need a unanimous three-vote check. Anchors that nobody asked about get written questions. On a 120-item audit, 95.8% were valid by majority and 84.2% unanimously. Of 60 L2 rubrics, 90% passed by majority.
Fourteen setups are compared. Full context reads everything before the cutoff, using Claude Sonnet 5, GPT-5.6-terra, DeepSeek-v4-flash, or Qwen-3.5-9b. Retrieval is BM25, dense top-20, GraphRAG, and RAPTOR. External memory is Mem0, LightMem, A-MEM, MemoryOS, xMemory, and EverMemOS. All of those use GPT-5.6-terra as the task model. The judge is GPT-5.6-sol, and scoring stays in Chinese. On 1,152 rubric decisions the judge agrees with people 89.3% of the time; people agree with each other 89.8%. When the judge errs, it withholds credit and applies penalties too often.
Full context still helps best. DeepSeek-v4-flash scores 0.669 on L1 and 0.469 on L2, first on both. EverMemOS reaches 0.621 on L1, above the same backbone, GPT-5.6-terra, given the full history at 0.570, then scores 0.247 on L2 and falls from 3rd to 10th. Qwen-3.5-9b with the full history still scores 0.348 on L2, above every external memory system.
| Method | L1 Atom Recall | L2 rubric |
| DeepSeek-v4-flash, full context | 0.669 | 0.469 |
| Claude Sonnet 5, full context | 0.628 | 0.428 |
| EverMemOS | 0.621 | 0.247 |
| GPT-5.6-terra, full context | 0.570 | 0.431 |
| BM25 | 0.539 | 0.370 |
| Qwen-3.5-9b, full context | 0.494 | 0.348 |
| xMemory | 0.546 | 0.203 |
| LightMem | 0.227 | 0.139 |
The order flips across the two tasks. On L1, EverMemOS and xMemory (0.546) beat every retriever (0.504 to 0.539). On L2, every retriever (0.334 to 0.370) beats every external memory (0.139 to 0.269), and all of them trail full-context GPT-5.6-terra at 0.431. MemoryOS, best of the external systems on L2, scores 0.269. BM25 and dense retrieval both beat GraphRAG and RAPTOR. A deeper memory layout does not buy a higher score.
Harder dependencies cost points. Full-context GPT-5.6-terra falls from 0.685 on M1 to 0.377 on M3 for L1, and from 0.505 to 0.303 for L2. All 14 baselines lose L1 recall once the query leaves a single memory. EverMemOS loses less there, 0.679 to 0.561, but its L2 on the easiest slice is 0.137, against 0.284 and 0.253 on M2 and M3. The one fact is already at hand, and the reply still skips it.
The benchmark links 497 memories that an L1 question asks directly and an L2 request in the same group also needs. Full-context models write 73% to 85% of what they recalled into the reply. EverMemOS writes 65%. For six memory systems, recalled-but-unused pairs are 21% to 27% of all pairs, against 11% to 16% with full context. That conditional-use rate tracks the L2 ranking at 0.89. Recall itself tracks it at 0.56. Of the unused memories, 36% never entered the L2 input, and 64% were in the input and still unused.
Relational Evidence Refinement follows links among retrieved memories and fetches evidence the request never names. LightMem, which drops the most of it, moves from 0.139 to 0.202 on L2 and gains about 8.4 points of L1 recall. EverMemOS moves only from 0.247 to 0.265. Assistance Constraint Binding labels facts already in context by owner, scope, and whether they are still valid, then binds those conditions to generation. EverMemOS reaches 0.279, and xMemory moves from 0.203 to 0.242. Using both is best for each system, and the joint gain is smaller than the sum, because each unused memory falls under only one bottleneck. Dense retrieval costs $55.0 across all 2,395 questions, against $260.4 for full context on the same backbone. EverMemOS costs $814.9 and needs 11,138 questions to break even. LightMem costs $69.6 and finishes last on both levels.
Choosing a group memory from the recall board chooses the worse assistant. EverMemOS already beats the same backbone reading the whole history, then loses to BM25, and to Qwen-3.5-9b with full context, once the output is an open request. Hierarchical memory keeps evidence that spans people and turns. It does not turn that evidence into a constraint the reply has to obey.
While the history still fits, full context is the ceiling for help. It feeds 52,435 tokens on average. EverMemOS feeds 21,123. RER plus ACB lifts EverMemOS to 0.279, still under 0.431 and under BM25 at 0.370. The patch targets the two real failures. It does not change who should answer the group. The saving is context and the bill.
The chats come from one product: 50 Chinese groups about ordinary life, not English workplace threads and not more adversarial rooms. Raw logs will not be released. The public set, after acceptance, is rewritten and pseudonymized. The same model decides whether the history changed the answer, so the kept requests can tilt toward what that model treats as useful. The paper says generative tools were used in filtering, cleaning, and method design.
M3 has only 300 items. The main text does not split questions members actually sent from questions written later to cover unused anchors, and the later wording may not sound like a real call to the assistant. The judge, GPT-5.6-sol, is the same family as the task model, GPT-5.6-terra. It ranks systems as people do, and it scores low. Style bias is not tested on its own. RER and ACB are attached only to three memory systems. On the strong ones, L2 rises by about one to four points and still misses full context. Group histories average 98,500 tokens and run to 972,800. Whether full context was truncated on the longest groups is not reported separately. Flat, planar, and hierarchical memory is not an ordering of ability.