MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation
Ryuichi Sumida, Koji Inoue, Tatsuya Kawahara
EMNLP 2026 (Main Conference)
cs.CL, cs.HC
2026-08-25
Adding memory capacity lifts Direct QA from 19.7% to 70.1% with no satisfaction gain; the same system answers 78.8% of facts when asked and cites 7.9% of them in conversation.
Long-term companions are built on a quiet assumption: if the model remembers prior chats, users will be happier. Memory benchmarks almost all test that memory the same way. They rewrite history into fact-seeking questions (Direct QA): which artist did the system recommend last time? LoCoMo, LUFY, and RealTalk all sit in this format.
Nobody checked whether a higher QA score moves real ratings. Being able to recall a fact when asked is a different skill from noticing a cue and weaving that fact into a reply.
Kyoto University ran a 4-month field deployment. Forty fluent English speakers talked daily with Luke, a diary companion on GPT-4.1-mini: 1,872 sessions, 21,575 turns. Each session was randomly assigned one of seven memory conditions. Every condition already had a summary of prior chats; the experiment tests extra capacity on top of that summary.
Importance is scored by a RoBERTa model finetuned on LUFY. Users rated each session 1-7. After dropping 11 users with near-constant ratings (SD ≤ 0.5) or low-quality content, 1,270 sessions from 29 users remain, with satisfaction z-scored within user.
GPT-5.4 then scanned the corpus for moments where memory was actually at stake. Human review kept 147 true positives at 95.5% precision. These moments are rare: about 3.5% of sessions and 1.4% of user turns, one every 73 user turns. The 72 user-cued cases (62 re-provisions plus 10 memory probes) become MemUse, split into 316 fact questions. The same reconstructed context is scored three ways:
On a 56-item subset, human-human Cohen's κ on Natural Integration is 0.57; the judge's positive rate matches humans (51.8% vs 51.8% / 55.4%).
Capacity lifts existing-benchmark Direct QA from 19.7% (Summary) to 70.1% (LC-100%). Satisfaction does not follow. Every condition sits within 0.06 within-user SD of Summary; Summary is slightly ahead (+0.02 vs -0.04 for LC-100%). Response length, proper-noun rate, and cross-session continuity each predict ratings. Memory condition does not. Latency is not the confound: mean latency 1.93 s on Summary versus 3.27 s on LC-100%, and |ρ|<0.06 against ratings.
| Condition | Existing QA | MemUse Direct QA | Reference | Natural Integration | Sat. z |
| Summary | 19.7% | 44.9% | 7.9% | 23.6% | +0.02 |
| LC-100% | 70.1% | 78.8% | 7.9% | 22.2% | -0.04 |
| RAG-100% | 63.7% | 62.2% | 7.3% | 26.4% | -0.02 |
Restrict the lens to the 48 sessions where users cued memory, and the picture flips. Natural Integration tracks satisfaction (ρ=+0.29, p=.046), +0.56 SD when it succeeds. Direct QA does not (ρ=+0.03). LC-100% ranks first on Direct QA (78.8%) and last on Natural Integration (22.2%). When users have to re-supply lost context, longer utterances predict lower ratings.
Hold model and context fixed: MemUse Direct QA is 78.8%, Reference is 7.9%, a 71-point gap. Per-instance Spearman between the two is ρ=-0.009. Of 72 LC-100% replies, 38.9% are generic empathy that ignores the cue, 23.6% hallucinate recall, 26.4% catch only the gist, and 8.3% fully integrate named details.
GPT-5.5 and Gemini 3.1 Pro repeat the shape. Direct QA rises 32-34 points with capacity; Natural Integration moves at most 8.2 points inside any model. Stronger models raise the floor (Summary NI 23.6% / 32.9% / 53.4%) and still do not turn extra context into extra integration. Mem0 and Letta lift Natural Integration to 58.3% and 56.9%, yet leave the fact gap open (41.5% vs 7.3%; 61.4% vs 10.8%). In a two-step extract-then-generate pipeline, 37 of 48 cases where extraction found the ground-truth details still failed to mention them, 77%. The bottleneck is conversational generation, downstream of retrieval.
System-initiated callbacks have no matching reward (ρ=-0.05). Users continue the recalled topic in 30% of 64 cases. Eight mistimed callbacks sit 0.48 SD below the user's own mean.
This is a measurement warning. Direct QA scores whether a fact can be recovered when asked. Users score whether the reply picked up the cue they just dropped.
Stacking context and RAG makes the first number look great and leaves the second almost flat. LC-100% costs 22× input tokens versus Summary and 3.27 s versus 1.93 s mean latency, with a slightly worse rating. Capacity is the expensive knob that does not buy the behavior that tracks satisfaction.
Mem0 and Letta show a different memory architecture can raise integration offline. They do not close the fact-level gap, and this paper does not claim those gains would move ratings in deployment. The next useful work is on the generation side: when to name a detail, and when to stay quiet.
All seven conditions include the same summary, so the null is about extra capacity on top of a summary, not about memory in general. MemUse only flags explicit linguistic cues; the 3.5% session rate is a lower bound. The integration-satisfaction link is observational, 48 sessions from 16 users; the mixed-effects p-value is .082, and the Spearman result does not survive Bonferroni. The sample is 92.5% female and the setting is a diary. Natural Integration is a binary judge.