Memory Is a Derivation: The Distributed-Evidence Paradox in Long-Term Agents
Hongjun Liu, Chen Zhao
cs.AI
2026-09-29
DerivAudit audits agent memory writes: expanded pre-write history rescues ~60% of citation-insufficient memories, but 59–89% of unsupported writes still get admitted.
Long-running LLM agents compress past interactions into persistent memory, and later tasks treat what was stored as established fact, often combining several memories into new conclusions. The write step is a reliability boundary: once an error enters the store, the original conversation is long out of context, yet the error keeps shaping behavior.
Two failure modes look identical to a naive checker. A valid memory may carry citations that cover only part of its evidence, so citation-only verification rejects it. And individually supported facts can be composed into a stronger statement the history never licensed, adding a causal link or turning a plan into an accomplished fact; in one real case, the history says the user aspires to found a sports academy and the stored memory says she is actively building one. The authors, at NYU, treat persistent memory as derived state, with a requirement they call semantic derivation integrity: what gets written must be supported by the interaction history available at write time. Earlier reliability work (LongMemEval, HaluMem, Eywa, MemTxn) tests recall or manages provenance and commit control; none jointly examines evidence beyond attached citations, compositional meaning, admission decisions, and downstream effects.
DerivAudit audits 200 unedited writes each from LoCoMo and HaluMem-Medium, with Qwen3-30B-A3B, Gemma-4-31B, and Llama-3.3-70B-Instruct as verification models.
Citations alone leave roughly half of memories unsupported: 42.5% on LoCoMo, 51.0% on HaluMem-Medium. Broader pre-write history recovers 60.0% and 59.8% of those cases, lifting overall supported fractions to 79.5% and 77.0%; 17–21% of writes remain unsupported after expansion.
Admission replay (313 valid, 78 unsupported or contradicted):
| Model | Unsupported admission (cite / expanded / obl.-aware) | Repaired retention |
| Qwen | 84.6% / 80.8% / 76.9% | 82.1% / 93.8% / 92.9% |
| Gemma | 66.7% / 76.9% / 59.0% | 58.0% / 95.5% / 92.0% |
| Llama | 83.3% / 89.7% / 88.5% | 85.7% / 98.2% / 97.3% |
Expanded history consistently improves retention of valid memories with incomplete citations (+11.6 points on Qwen, p = 0.007). Rejection of unsupported writes is unstable: Qwen's drop is not significant on the 78 negatives (p = 0.24), and expansion alone makes admission worse on Gemma and Llama.
Swapping verification representations does not fix it either. On the 512-record suite, atomic claims detect only 16.5% of unsupported relations, predicate–argument QA detects 39.2% but retains only 56.4% of valid multi-passage memories, and the compact-relation view (59.5% detection, 94.9% retention) is the most balanced; no view wins on both ends across models.
When evidence is distributed across passages, holistic and obligation-aware verification detect only 8.6% and 11.4% of invalid Qwen cases. This is the distributed-evidence paradox: valid memories often need evidence combined across interactions, yet the same distribution makes unsupported compositions harder to reject. Controls pin the mechanism: replacing plausible premises with token-matched irrelevant content drops invalid acceptance from 65–88% to 0–10%, so the failure is plausible-premise composition bias, not context length.
Finer checking has its own failure mode. Adding four obligations that are independently true lowers Qwen's median support score by 7.7 log-odds units (98.7% of records move down) and collapses admission from 93.3% to 2.6% on Gemma and from 97.4% to 35.5% on Llama; the paper calls this accumulated verification noise. Downstream, a distorted memory raises failure by 83.8–93.0 percentage points across the three readers, and removing a needed valid memory raises it by 92.8–99.5 points.
Three takeaways for memory-system builders. Citation-only verification systematically rejects good memories: about 60% of citation-insufficient writes are actually supported, so tighter gates mainly lose information. Expanding evidence fixes recall, not rejection; unsupported admission sits between 59% and 89%, and neither representation nor aggregation changes that dependably across backbones. More detailed checking is not free safety: four extra true checks alone collapse Gemma's admission to 2.6%, which matches how granular verifiers often behave in deployment, as mass rejection. Since both error types cost roughly 90 points downstream, 'reject when unsure' is not a safe default either. The audit harness itself, from paired evidence views and obligation decomposition to admission replay and downstream probes, can be lifted onto any memory pipeline.
Author-stated: reference labels come mainly from two model instances with blinded adjudication (stratified human review agrees 96.0%, but these remain model labels); expansion is retrieval-bounded, so unrecovered support does not prove invalidity; natural data contain few contradicted cases and few explicit relations; results vary with model, view, prompt, and aggregation rule; the downstream probes are not a closed-loop memory system.
More caveats from a close reading. Seventy-eight negative records cannot power the admission statistics, and the paper itself flags Qwen's reduction as unresolved. Only three open 30B–70B models were tested; whether a frontier closed model still admits 59–89% of unsupported writes is unknown. The 60% rescue rate depends on the specific BM25 top-12, 16-passage budget, so a different retriever would move the number. And the reference labels come from the same class of models being evaluated, so circularity is reduced, not eliminated.