RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations
Arman Behnam, Sunglyoung Kim, Liangwei Yang
cs.AI
2026-10-01
Quis Lab's RealCompanion releases 10 real companion logs, 27,218 messages over up to 120 days: only 3.4% of turns need the past, a median of 2,157 messages back.
Memory benchmarks such as LoCoMo and LongMemEval write the long chat around the fact they will later ask for, so almost every turn depends on the past. PersonaChat through PersonaMem state a persona and then ask about that statement. The past is defined as always relevant, and who the person is gets fixed before any system reads a word.
Months with a companion split into two jobs. One is who this person is, which stays mostly stable. The other is whether the sentence in front of you needs something said earlier, which changes on every turn. In Figure 1 the person had said they were unsure about managing people and applied so they would not keep wondering. Congratulating them on getting exactly what they wanted uses a judgment the conversation does not contain. Both jobs can be checked only against what that person said. The logs are private and often touch health and family, which is why public sets have used invented people.
RealCompanion, from Quis Lab, releases ten consented relationships from one companion product's production store. Spans run from 36 to 120 days, and the corpus holds 27,218 messages. The two heaviest relationships hold 73%. Each person ships as five files: the conversation, a profile of facts they stated, a persona of how they think, feel and decide, chat labels with the reasoning trace behind them, and a question set.
Three tracks. Reconstruction reads one full history and rewrites the profile and persona. Chat probes are the user's own messages. Questions are written from the profile: 3,235 scoreable items, plus 63 designed to have no answer.
Chat labels are derived in five stages, with the decision and the reason written in the same call. Stage A names what the probe points at. Stage B checks the source and may only drop a citation, so a miss can only make memory look rarer. Stage C applies one of nineteen rules. Recent turns alone are basic, a profile claim is intermediate, an earlier exchange is hard, and an empty context is edge. Stage D writes the reference reply from the recorded context only. Stage E demotes or withdraws the item if that reply is not supported.
The proportional stratum is a random sample of user turns, 1,169 scoreable items, and the only slice that estimates a rate. The enriched stratum, 364 items, is a census of deep-history cases and is not a rate. The abstention stratum is 434 openers that should use nothing earlier. Of 1,533 scoreable items, 404 point outside the current thread: 223 profile claims, 116 earlier exchanges, and 65 past turns from the companion. On 167 of them the reference reply actually uses the citation.
Retrieval compares a random ranking, a recency window, the user's own recent messages, BM25, and an oracle that returns the gold set. A broken oracle returns messages outside that set and should score 0. Every message in the ablation is cut to 220 characters. C0 is the probe, C1 the three preceding turns, C2 the turns the label names, C3 ten from BM25, and C4 the ten preceding turns. C3m and C4m share one neutral heading and chronological order, so the word memories is not part of the comparison.
Rates use the proportional stratum only. Under the recorded reading, 3.4% of sampled messages need earlier material (2.5% to 4.6%). The strict reading drops referents already visible on screen and lands at 1.3% (0.8% to 2.1%). A census outside the sample gives 3.2% and 1.73%. Three of the ten people have no memory-bearing probe under the looser reading, and five have none under the stricter one. Verification can delete a citation and cannot add one, so these rates are lower bounds.
When memory is required, the furthest needed message sits a median of 2,157 back, with an upper quartile of 5,114 and a maximum of 12,542. The nearest needed message still has a median distance of 450.
| Method | Distant hit@5 | All probes with a reference |
| Random | 0.005 | not given |
| Recency | 0.022 | 0.959 |
| BM25 | 0.248 | 0.236 |
| Oracle | 1.000 | not given |
| Broken oracle | 0.000 | not given |
Hit@5 is how often a required message appears in the top five. Among 1,477 probes with a reference set, 72.6% are answerable from the current thread, so recency hits 0.959 overall, 0.022 on distant items, and 0 once on-screen referents are removed. BM25 stays near 0.24 on both cuts and still misses about three quarters of the distant items. Averaged by participant, its distant hit@5 is 0.424 against 0.248 pooled by item. A pooled score follows the mix of the corpus. Coverage of the required message is 0.109 at 8k tokens, 0.453 at 128k, and 1.000 at one million.
Content match runs from 0 to 2. Three preceding messages score 1.622 and the recorded messages score 1.763. The gain is +0.140 overall and +0.108 on the proportional stratum. Probes that need memory contribute +0.004 of that gain, probes that do not contribute +0.104, and 96% of it is not a memory effect. On the 167 items whose reply depends on the citation, the same swap is +0.455. Before headings are matched, BM25 at 1.426 ties no context at 1.431. After heading and order are matched, ten BM25 messages still score 0.189 below the ten preceding ones.
A detector that reads the probe and three preceding messages is close to chance. AUROC is 0.530 at the natural rate, 0.548 on cold opens, and 0.601 against its own target, where the threshold catches 6 of 40 and raises 40 false alarms. Shown three retrieved messages, a detector reaches 0.802 on authored questions and 0.546 on chat probes. With no context the misfire rate is 0. With context it rises to 29.2% when the messages were selected for the probe, and to 60.8% when ten are supplied with no regard to the probe. Labeling those same ten as retrieved memories, instead of earlier turns, raises use by 10 to 14 points on every population tested, including probes that need no memory. Relabeling a recency window does not.
| System | F1 | Precision | Recall | Tokens | Run agreement |
| Claude Opus 5.5 | 0.686 | 0.557 | 0.893 | 546M | 0.935 |
| Codex GPT-5.6-sol | 0.688 | 0.573 | 0.861 | 18M | 0.932 |
| Antigravity (Gemini 3.8 Flash) | 0.701 | 0.591 | 0.861 | 36M | 0.946 |
Three agent systems, three runs each, reconstruct the persona at F1 0.69 to 0.70, with about a 30-fold gap in tokens processed. They agree with one another at 0.86 to 0.88, above their overlap with the released file. Precision of 0.56 to 0.59 and recall of 0.86 to 0.89 come to about seventy or eighty extra fields for every hundred they recover.
Memory layers now shipping were tuned where the question already names what to retrieve. Restraint is vacuous there, because every item was built to need the past. On real traffic the rate is about 3%. A system that looks backward on every turn is wrong mainly on the turns where it should have stayed with the recent thread.
A pooled gain in reply score has the same blind spot. On the proportional stratum, 96% of the gain from supplying the recorded evidence comes from probes that need no memory. One headline number mostly measures how large that no-memory majority is. Long-range retrieval has to be scored on the distant subset, or recency at 0.959 against 0.022 looks as if it had already beaten BM25. Calling a block of history memories adds 10 to 14 points of use by itself. On personas, a frontier model and a flash-tier model tie. About 30 times the tokens does not buy fewer invented traits.
The release is ten relationships on one product. Two people contribute 19,870 of the 27,218 messages, and bootstrap intervals over ten people are wide. Empathy, forecasts of later behavior, and values are not scored.
The 3.4% rate is a lower bound. A missed referent makes memory look rarer, and the census does not show that stage A caught every one. The released wording was rewritten by a model. The words changed, and the events and context were meant to stay. Questions, profiles, personas, and chat labels came from a model pipeline and were then reviewed by hand. The profile audit records 1,471 supported, 1,025 repairable, 455 unsupported, 390 to review, 37 supported after pruning, and 202 not judged. Agreement with humans is κ 0.804 on a binary verdict and 0.545 on the graded scale. A cheap guess made before verification disagrees with the final tier on 685 of 1,533 items.
The companion that produced these histories had a memory of its own. On 22.0% of the 404 memory-bearing rows, the evidence was sent entirely by the companion. After removing rows whose task is to recall what the companion said, 7.1% remain, 24 of 339. Those gold facts may be the system telling the user something, then the benchmark scoring it as something the user revealed. C1 and C2 also use different headings, and that slice of the gap was not isolated. Ethics review was the operator's own committee, not an independent IRB. The release forbids profiling, identification, and commercial use.