SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue
Haobo Zheng, Tan Tang, Yan Chen, Weijie Wang, Yingcai Wu
cs.CL, cs.AI, cs.IR, cs.LG
2026-09-23
SpeakerMem-R1 stores speaker-labeled messages beside person- and group-level state. Accuracies: 47.9%, 69.2%, 61.9% on three benches; RL lifts a 3B Writer from 57.38% to 68.20%.
Long-term conversational memory for two-person chats is mostly chunk, compress, retrieve by relevance. Group chats add speakers, reply structure, parallel topics, and later revisions. General-purpose memory systems drop who said what, whom a claim is about, and whether a fact is personal or shared, and they struggle to rebuild current state from interleaved clues. On GroupMemBench, SocialMemBench, and EverMemBench, BM25 or dense retrieval can match heavier frameworks. The bottleneck is attribution and state reconstruction, not more text in the candidate set.
SpeakerMem-R1 keeps two tracks. System 1 stores each message with text, speaker, time, and channel, with no language model. System 2 is four derived layers: person-scoped Core (stable identity, facts, stances, recurring behavior) and Profile (observations about a person, including cross-person beliefs), plus group-scoped Interaction (cross-speaker events, relations, decisions) and Insight (norms, consensus, exceptions). Each derived record splits source from owner: "Alice thinks Bob already agreed" has source Alice and owner Bob. Updates never overwrite; they append a node and link via supersededby, so queries can ask for the head or the full chain.
At query time the question is compiled into constraints: which person or group rows, which event, head versus full history, and source-owner filters. The two tracks have separate budgets. System 1 retrieves wording and nearby context, with one optional extra search. System 2 expands person and group rows, then filters by event and time. An empty derived row falls back to that person's verbatim messages; if still empty, the row stays empty instead of borrowing someone else's records. Evidence is composed along people, relations, and update chains for a frozen answerer.
Only the Writer is trained. Qwen2.5-3B emits Add, Update, or Noop. SpeakerLevenshtein matches one-to-one inside owner buckets and penalizes the worst owner, so frequent members cannot hide missing people or GROUP. A global term is QA gain of System 1+2 over System 1 alone, so questions already solvable from verbatim text are not credited to the Writer. Eight trajectories per conversation, group-relative advantages, clipped GRPO; retrieval and answering stay frozen.
Taking the best SpeakerMem-R1 configuration per benchmark: GroupMemBench 47.9% versus BM25 44.6%; SocialMemBench 69.2% versus A-MEM 56.8%, near Full context at 69.4%; EverMemBench 61.9% versus BM25 52.5%. On the public EverMemBench board (GPT-4.1-mini answering, Gemini-3-Flash judging), SpeakerMem-R1 scores 62.33% (1,496 / 2,400) against EverOS at about 60.08% and RippleMem at 54.75%. Temporal questions reach 35.00% versus EverOS 20.33%; multi-hop, skill, and role attribution stay weak.
| benchmark | SpeakerMem-R1 | strongest mainstream |
| GroupMemBench | 47.9% | BM25 44.6% |
| SocialMemBench | 69.2% | A-MEM 56.8% |
| EverMemBench | 61.9% | BM25 52.5% |
| EverMem public board | 62.33% | EverOS 60.08% |
Removing either track hurts all three benchmarks; person-only or group-only views also drop. On SocialMem, turning on extra retrieval (ASK) moves accuracy from 69.2% to 64.9% while token-F1 rises, because extra people can void an otherwise overlapping answer. On two-person LoCoMo, accuracy is 70.85% of 1,986 questions; multi-hop and open-domain remain weak.
On 10 held-out SocialMem networks (305 questions) with query and answer frozen, a Qwen2.5-3B SFT Writer scores 57.38%; 30 steps of RL reach 68.20%, against an LLM writer at 71.48%, or 95.4% of that reference.
Group-chat failures are concrete: wrong speaker, wrong scope, stale state. The design keeps checkable wording next to person- and group-sliced state, then composes them at query time, instead of stacking another graph or summary. The Writer can be a local 3B model with retrieval and answering frozen. Gains are task-dependent: near Full context on SocialMem, only 3.3 points over BM25 on GroupMem.
The system assumes a reliable roster, identifiable speakers, and usable timestamps. Aliases, membership changes, implicit audiences, and parallel events remain hard. Cross-evidence reasoning, knowledge outside memory, and open-domain QA are documented weak spots. The RL gain is a 305-question controlled protocol, not cross-domain generalization. ASK hurts SocialMem accuracy, so another retrieval round is not free. LoCoMo multi-hop is 41.13%; dual-track memory does not automatically fix two-person multi-hop.