VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction
Zhifei Xie, Jiaqi Lang, Ze An, Yifan Zhao, Dongchao Yang, Kai Li, Ziyang Ma, Mingbao Lin, Chunyan Miao, Shuicheng Yan
eess.AS, cs.AI, cs.IR, cs.MM, cs.SD
2026-08-27
VoiceMem adds streaming dual-brain memory to speech: facts left, persona right. Top-5 retrieval takes 134 ms and scores 76.39 vs Mem0's 52.27.
Spoken dialogue models can already listen and talk at the same time. Their memory stacks still behave like text agents: retrieval often takes 2-3 seconds and dumps a top-100 list into context. That blows a 500 ms turn budget and swamps what a speech model can read. Affect is usually a retrieval-side extra, not a store that can tell a stable trait from a reaction to one person or event.
VoiceMem treats memory as infrastructure for real-time speech. The design target is dense top-5 retrieval, latency hidden inside the VAD silence window, and parallel upkeep of facts versus persona.
The system is two-layer. The upper layer is a dual-brain index; the lower layer is a swappable memory engine, Mem0 in the paper. The left brain stores facts in a schema-entity index: schemas route coarsely, entities point at people, events, and concepts. While the user is still speaking, a matcher hits schemas and entities on the partial transcript, expands one hop, then searches only that pool. When a cluster gets too diffuse, an emergence step promotes a subgraph that is repeatedly retrieved together, after an LLM checks relevance, importance, and completeness.
The right brain stores who the user is. Independent nodes hold stable dispositions. Cross-entity nodes pin affect to a left-brain entity, so anger at one coworker is not written as a personality trait. Short-horizon attribution updates within a turn; long-horizon attribution consolidates recurrent evidence after a session.
Latency is hidden in four stages: match during listening, expand the graphs after 200 ms of silence, run the backend search between 400 and 500 ms. A typical VAD waits 500 ms; the dense search itself is 134 ms. To teach speech models to use the memory, the authors run black-box on-policy distillation from closed-source teachers, producing ChatMem-400K and a human-curated ChatMem-Bench covering information, persona, affective attribution, and paralinguistics plus environment.
On three information-memory suites VoiceMem averages 76.39, against 52.27 for Mem0, 65.83 for MemOS, and 60.49 for full context. On LoCoMo at K=5 it scores 91.2 with 430 memory tokens and 134 ms; EverMemOS needs 1,899 tokens to reach 83.13. Latency stays near 134 ms from K=3 to K=100 because routing bounds the pool before ranking. Removing the upper index costs 9.9 points at K=5 on LoCoMo.
On three persona suites the GPT-4o-mini setup scores 74.16 versus MemOS at 72.27 (+1.89). The fine-tuned responder reaches 76.56 (+4.29 vs MemOS). ChatMem-Bench has 316 questions over 53 hours of audio: 68.73 versus 53.95 for MemOS. The gap is widest on acoustic categories, where text systems sit between about 3 and 27 while VoiceMem reaches 45 to 54. Dropping the same index onto LangMem and Zep lifts LoCoMo by 15.76 and 22.92 points.
Teams building duplex voice agents get a routing layer they can sit on top of an existing memory engine. They do not have to open the budget to top-100, and they do not have to add seconds inside a turn. Persona is a graph with targets, not a similarity bonus. The backend is replaceable; three stores already pick up the same index.
This is not a new end-to-end speech model. It is memory plumbing. The default path still runs on ASR text, with audio attached as optional nodes.
There is no dedicated Limitations section. The ChatMem-Bench annotation protocol is deferred to a follow-up report, so the 316-question split is hard to reproduce. The persona headline of +4.29 uses the fine-tuned responder; GPT-4o-mini is only +1.89 over MemOS. The "nearly 30 points versus Mem0 at top-200" claim is not the same comparison as K=5 versus K=5, where LoCoMo is closer to +24.
The pipeline assumes a 200 ms pause before reply. Full-duplex barge-in can make the prefetched query disagree with the final transcript. Right-brain consolidation is a session-level LLM pass; long-term drift is unmeasured. Training conversations are teacher-generated in synthetic memory worlds. Real noise, dialect, and overlapping speakers are not in the reported numbers.