Route to Remember: Adaptive Memory Selection for LLM Agents
Zeyu Fang, Jun Yin, Yongsheng Mei, Xu Han, Chaitali Joshi, Zhongkai Sun, Jie Hao, Bin Guo, Chengyuan Ma, Benjamin Z. Yao, Chenlei Guo
NeurIPS 2026 Workshop PALM
cs.AI
2026-11-04
Amazon and GWU train a per-user LoRA with on-policy self-distillation: 59.9% on HaluMem beats every text-memory baseline at a fraction of the per-query read cost.
Production agents keep long-term memory as text: extract key points from sessions, store them, retrieve into the prompt at query time. Every query pays retrieval latency and context tokens again, the bill grows with accumulated history, and extraction quality caps accuracy. Writing memory into model parameters flips the structure: read cost stops depending on dialogue length, and the write is a one-time training pass. Earlier attempts at parametric memory dropped sharply once context was removed at inference, and the closest prior diagnostic concluded that a per-user adapter and retrieval are complementary, with neither dominating. This paper (Amazon and GWU, NeurIPS 2026 PALM workshop) asks directly how far a parametric store can be pushed on episodic and semantic recall, and what the bottleneck actually is.
Two pieces: OPSD writes the memory, VtR routes the read.
On the data side, each session is converted by the base model into timestamped memory points, then expanded into three kinds of QA pairs: paraphrases of the same fact, counterfactuals that state it wrongly and demand correction, and unknowns about attributes never recorded, where the correct answer is a refusal. The unknowns turn out to be the sole source of the model's willingness to decline; removing them collapses the boundary category from 88% to 1%.
In OPSD (on-policy self-distillation), the teacher is the base model conditioned on a recency-bounded window of the user's memory points; the student is a per-user LoRA on the same base model, answering with no context at inference. Training runs on the student's own distribution: the student samples an answer, the teacher scores exactly those tokens, and the loss is the KL divergence between them. An off-policy SFT variant suffers exposure bias and degenerates into repetition loops. Two counterintuitive choices matter. The teacher must stay the frozen base model; swapping in the accumulating student snapshot compounds errors across sessions and collapses the late-horizon forgetting curve, worth about 14 points. And the question pool must not keep expanding: adversarially generated questions drift onto unrelated topics and dilute the facts worth retaining.
VtR ignores the query and audits the answer. The parametric memory answers first, retrieval runs twice (once with the question, once with the answer), a cheap three-way verifier labels the answer supported, conflicting, or unsupported, and only failed answers are regenerated by the base model without the adapter. Anchoring one retrieval on the generated answer is deliberate: a plausible answer maps the question into the semantic region where memory statements actually live, a better-posed matching problem than question against statement.
HaluMem native questions (n=1438, main model Qwen3.5-9B, Claude Sonnet 4.6 as judge):
| System | Overall Correct% |
| GT points (oracle upper bound) | 64.0 |
| Raw dialogue in context | 51.3 |
| Extracted memory in context (the teacher) | 51.9 |
| MemOS retrieval | 47.8 |
| SFT on identical data | 36.9 |
| OPSD | 59.9 |
| OPSD + VtR | 66.1 |
OPSD leads retrieval by 12.1 points (user-clustered CI [9.4, 14.8]), positive for all eight long-horizon users (7.1 to 18.1 points); dropping the boundary category widens it to 18.0, and the ranking holds at 4B, 9B, and 27B. Costs invert: roughly 150M write tokens paid once, then 326K input tokens read across the entire evaluation, four orders of magnitude below full context and five times below retrieval. Break-even is about 600 queries against full context, on the order of 10^5 against retrieval.
Retention separates the paradigms. Context methods fall off a cliff when the session leaves the window (raw dialogue by age 40, extracted by 60); weight memory decays smoothly from 59.9 to 32.4 by age 80; retrieval is nearly flat, 47.8 to 40.1, overtaking weights past roughly twenty sessions. Inside weight memory the refusal habit survives (boundary 81.7 to 88.1) while the content it protected collapses.
Routing is where everything failed: query similarity, a learned query classifier, and answer confidence all fail to predict whether the weight answer is correct (AUC 0.516); a length-constant probe detects answerability at 0.918 but is at chance on correctness. A per-question oracle would reach 69.3%; VtR captures 6.2 points of that. The ranking reproduces on LoCoMo (OPSD 46.9 vs extracted 46.4, SFT 16.9, VtR 60.4), where every single paradigm sits at a 2.5 to 10.6 floor on adversarial questions and VtR lifts them to 73.1.
The most consequential number is not 59.9 but the 23-point gap between OPSD and SFT on identical data. The gap is entirely the objective. Early pessimism about weight memory rested almost entirely on supervised fine-tuning; the medium was likely never the constraint, the recipe was. The deployment recipe is concrete: distill recent, recurring, widely shared knowledge into weights, leave the long tail and absence detection to a text store, and compose the two by verifying answers rather than routing on the query. The read-cost profile suits knowledge written once and queried many times. That said, this is a research setup with one adapter per user; the paper does not measure what it costs to serve adapters for thousands of users.
The authors' own list: high write cost; gradual forgetting of superseded facts; nothing to inspect inside parameters, so unrecorded attributes get confabulated with specific invented salaries and body weights; VtR fixes the boundary category but reduces memory-conflict accuracy (76.0 to 66.1); the advantage concentrates on recent material. Why the student beats its own teacher (59.9 vs 51.9) gets three hypotheses and no verification. About a fifth of boundary hallucination verdicts are valid abstentions scored wrong, so the rubric is noisy, and a single judge with no partial credit compounds this. Open questions beyond the authors' list: eight long-horizon users is a small evaluation, HaluMem is synthetic, and VtR adds roughly 20M write tokens, which only pays off under heavy query load.