Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability
Ankit Goyal, Jaideep Ray
cs.AI, cs.CL, cs.IR
2026-09-05
On 48 synthetic histories, KG-fixed moves +0.0004 after a writer swap; NOTES drop 13.28 pp one way; a 50/50 mixed index keeps only 4.96 of an 11.90-point embedding gain.
Agent models rotate every few months. Memory stores can live for a year. After an upgrade the notes are still there, the vectors are still there, and the service starts cleanly. The new model may still misread another model's notes, mixed embedding versions can break retrieval without a dimension error, and facts dropped during compression cannot be rebuilt from the notes alone.
That is a different question from "is the new model smarter." The engineering question is how much each common memory format loses when the writer, the embedder, or the repair path changes, and where in the pipeline the loss happens.
Two LinkedIn authors freeze the same history and write it in four formats, then change one component at a time:
The testbed is 48 synthetic histories with 160 questions each, randomized answer codes, exact match, and no LLM judge. The reader/writer pair is Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct-1M. Embeddings move from bge-large-en v1.0 to v1.5. Both emit 1024-dimensional vectors, so a mixed index stays up.
The headline metric is RPAS: how much of a new reader's own-store score remains when it inherits the old writer's store. Repair uses CTR: at matched budgets, does rebuilding from raw history or rewriting the store alone get back to 90%/95%/99% of own-store accuracy. Four tests were locked in a signed Git tag, hypothesis-lock-v1, with a five-point action threshold.
A fixed schema barely cares who the writer was. KG-fixed accuracy moves by +0.0004±0.0020 after a writer swap, with own-store scores already high (Llama 0.846, Qwen 0.988).
NOTES is directional. Averaging the two ways hides the failure. Qwen reading Llama notes drops 13.28 points versus Qwen's own notes (0.472 to 0.339). Llama reading Qwen notes gains 9.91 points (0.376 to 0.475). In calibration, Qwen kept 85.6% of required evidence in about 143 KiB; Llama kept 64.5% while using about 159 KiB. Note quality is part of the memory design. The symmetrized tests H1 and H4a miss the 5-point threshold because the two directions cancel.
RAG already misses evidence before any reader swap. LC-RAW scores 0.712-0.911; RAG only 0.535-0.565, surfacing the right evidence about 60% of the time. Diagnostics attribute 81% of the RAG deficit (0.364 of 0.450) to retrieval, and 80% of the NOTES deficit (0.467 of 0.584) to construction. Restyling NOTES recovers nothing: Qwen rewriting Llama notes changes accuracy by -0.012.
A half-migrated embedding index is worse than it looks. The old v1.0 index scores 0.426. Full re-embedding to v1.5 reaches 0.545 (+11.90). A 50/50 mixed index only reaches 0.475 (+4.96), leaving most of the upgrade on the table. An oracle router sits at 0.596. Dimensions match, so nothing crashes.
Repair is harsher. Store-only NOTES rewriting never hits 90% of own-store accuracy in any of 48 histories at any tested budget. With raw history, Qwen recovers 34/48 at a median of about $0.76; Llama recovers 0/48 because every attempt hits the output-token limit. RAG re-embedding recovers 96/96 at about $0.013. Rebuilding KG-fixed into the shared schema recovers 91-96 of 96 at near-zero extra cost. H2 (mixed index worse than full re-embed, +6.95 pp) and H7a (raw repair beats store-only, +8.90 pp) clear the five-point threshold.
This is a migration checklist, not a new memory architecture. Free-form notes have to be tested writer-to-reader, both ways; averages wash out failures. Embedding upgrades need a separate index and a cutover, not two spaces poured into one table. Compressed memory cannot recreate omitted facts, so keep a controlled source copy when policy allows. When answers go wrong, check writing, retrieval, and reading in that order before swapping the reader.
The numbers belong to this synthetic workload and two 7-8B models. A portable schema here does not mean every knowledge graph is portable.
The KG-fixed schema was written from this workload's event fields, and its read path differs from NOTES, so the result does not say graphs beat notes in general. RAG is a single-stage dense retriever with no reranker; the 40% miss rate and the mixed-index penalty will move with the pipeline. Raw-history repair worked in only one direction, and the repair model is tied to the migration direction. Only two similarly sized open-weight models were tested, on scripted histories that fit both context windows. Longer natural conversations, larger models, and subjective questions may fail differently. The signed tag is an internal analysis lock; the bootstrap was run after the t-tests.