Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations
Wenxu Jia, Xize Cheng, Zihan Zhang, Dongjie Fu, Linjun Li, Wenshi Chen, Yangyang Wu, Tao Jin
cs.AI, cs.IR, cs.SD, eess.AS
2026-09-26
VoxPolyMem adds speaker ID, interaction-graph memory, and EG-GRPO retrieval for multi-party speech; it scores 85.0 on VoxPolyBench, 23.6 above the strongest baseline.
In-car, meeting, and home assistants talk with several people, then meet the same people again on later days. Long-term memory systems are still scored mostly on one-to-one text or image-text chats. Mem0, A-Mem, Zep, LoCoMo, and Mem-Gallery all sit in that box. H2HMem covers multi-party text. MSU-Bench covers multi-speaker speech. Almost no benchmark jointly tests cross-session voice identity, who spoke to whom, and whether a person's preferences actually changed.
ASR transcripts drop acoustic identity. Temporal graphs organize content without storing speaker and addressee roles. Retrieval is often a fixed recipe, so evidence that sits in another session, layer, or modality never gets a second look. The product is already multi-party speech. The methods and tests are still dyadic and mostly textual.
VoxPolyMem, from Zhejiang University and Meituan, has three parts: online speaker ID, a three-layer memory, and a trainable multi-round retriever.
Speaker ID uses ECAPA-TDNN embeddings matched by cosine similarity against an open set of anonymous voiceprints. A close match claims the turn; otherwise a new ID is created. The speaker count is not given in advance. Templates update by exponential moving average only on confident matches. After each session, fragmented IDs merge conservatively from accumulated voice evidence. Names and aliases are bound later by an LLM.
Interaction memory is a directed graph: people and messages as nodes, speaks and addresses as edges. Raw text, timestamps, images, and transcripts stay attached. Qwen3-VL-Embedding indexes content, separate from voiceprints. Fact memory stores self-contained statements with speaker, addressee, and time, taken from the current turn plus up to three previous turns. Profiles store name, aliases, voiceprints, background, and preferences.
Retrieval is sequential. A spoken query is first matched to a profile so the system knows who is asking. Each step picks a memory layer, a tool (dense, BM25, text-to-image, image-to-image), a rewritten query, and optional speaker or addressee filters. Reciprocal rank fusion (RRF) merges lists by inverse rank, then keeps at most 15 records. The model answers or continues, for at most three rounds.
EG-GRPO changes how Group Relative Policy Optimization assigns credit. At each state the old policy samples eight actions. Reward is coverage plus rank of gold supporting IDs that this action newly kept in context. Evidence already in hand scores zero, so the policy has to fill gaps. The answer model stays frozen. The policy is an 8B Qwen-VL model, trained for one epoch on about 900 evidence-annotated questions: 300 from Mem-Gallery, 400 from LoCoMo, 200 from VoxPolyBench.
Evaluation uses the new VoxPolyBench plus Mem-Gallery and H2HMem-Multi. VoxPolyBench has 18 scenarios, 176 sessions, 18.9 hours of synthesized speech, and 1,527 questions. Dialogues are generated by GPT-4.1 from event anchors. Corpus WER with Whisper large-v3-turbo is 2.14%. Every method reads the same whisper-turbo transcripts. VoxPolyMem runs its own speaker ID; baselines receive gold speaker labels. GPT-4.1-mini answers. GPT-4.1-mini and GPT-4.1 score independently and are averaged.
| Method | VoxPolyBench | Mem-Gallery | H2HMem-Multi |
| Strongest public baseline | M2A 61.3 | UniversalRAG 77.8 | AUGUSTUS 66.0 |
| w/o RL | 84.0 | 86.3 | 72.1 |
| VoxPolyMem | 85.0 | 89.6 | 74.4 |
The paper reports 85.0 on VoxPolyBench, 23.6 above the strongest external baseline; M2A's 61.3 is the highest external mean in the table. Weighted by question count, VoxPolyMem scores 86.6 against UniversalRAG at 68.1. Persona questions score 76.1 versus mem0 at 40.5. Interaction attribution scores 87.5 versus HippoMM at 61.3. Retrieval-and-reasoning (86.4) and memory evolution (92.1) also lead. The variant without RL already beats every external baseline, weighted 84.4.
Against that variant's 84.0 / 86.3 / 72.1, dropping fact and profile memory lands at 79.8 / 84.9 / 63.6. Dropping interaction edges cuts VoxPolyBench from 84.0 to 78.6. Single-round retrieval falls to 76.1 / 85.5 / 60.0, worse than dropping query rewriting (83.2 / 86.0 / 70.8). Search-R1, trained on final answer quality, drops VoxPolyBench from 84.0 to 81.3. EG-GRPO leads all three sets, with recall 87.4 and 93.6, and 1.2 to 1.3 retrieval rounds versus 1.4 to 1.8 for the controls. With reference utterance boundaries, speaker tracking hits 95.0% on IEMOCAP and 87.9% on AMI.
Meeting, car, and home voice products need speaker identity and addressee structure in memory. Dumping transcripts into a vector store does not cover that. The jump on persona (40-point band to 76) and attribution (60-point band to 87) sits exactly where the design spends its capacity.
Hierarchical memory plus multi-round retrieval, with no RL, already clears ten public baselines. EG-GRPO adds about 1 to 3 answer points and fewer rounds. The retriever is 8B. Extraction and answering depend on GPT-4.1-mini and synthetic data. Gains on public image-text sets are 11.8 and 8.4, much smaller than 23.6 on the in-house speech set, and part of that gap comes from questions written to test speakers and addressees. For an existing memory stack, this is a structure and a retrieval policy you can slot in, not a new backbone.
The paper is direct: generated dialogues and synthesized speech do not establish robustness to real accents, noise, or recording conditions. Voiceprints and profiles can enable tracking and disclosure. Deployment would need consent, access control, limited retention, and deletion. Leaderboard scores are not a license for surveillance or high-stakes identity decisions.
Corpus WER of 2.14% is far cleaner than real meetings. Baselines were given gold speaker labels, so VoxPolyMem won on a harder setting; a large share of the 23.6-point in-house gap is still that baselines have no interaction edges or profiles. Training used 200 gold-evidence questions from VoxPolyBench. The test split is described as held-out, but it is cut from the same generation pipeline. The same class of LLM writes the dialogues and questions, extracts memories, and judges answers, so the scores may sit high. A human audit is mentioned in an appendix without agreement numbers in the main text. The IEMOCAP and AMI figures assume reference utterance boundaries, not diarization from raw meeting audio.