MemoryAthena: Adaptive Routing over Latent and Generated Memories
Mingyuan Li, Guangsheng Yu, Juyuan Zhang, Xu Wang, Zhibo Man, Haonan Zhang, Shaoxiong Ji
cs.CL
2026-09-22
MemoryAthena patches frozen Engram readout only when generated residuals help. QA mean 37.65 to 39.28; six NLP tasks 76.73 to 79.13. Soft fusion and random routing lose to E-only.
External memory usually hands the model a stored object: retrieved text in RAG, a neighbor in kNN-LM, or a table entry in Engram-style systems. Prior work already splits that stack into addressing, storage, and reading, and can reuse a frozen table on a new backbone through an adapted reader. The residual that actually enters the network still comes from lookup.
MemoryAthena asks whether that residual can be generated. One generator (GE) conditions on retrieved Engram cues. Another (GH) never opens the table and builds a latent from clean causal hidden states. Direct Engram readout (E) stays available.
Generated memory is useful on some inputs and harmful on others. GE raises WebQA and loses on TriviaQA and HotpotQA. GH is weaker than E on every QA summary metric. Soft-mixing the three residuals drops the five-task mean from 37.65 (E only) to 28.62. The design problem is admission: when to intervene, which generator to use, and how hard to mix it into E.
The backbone is frozen Mistral-7B-v0.3. Memory is injected at layers 2 and 10. All three paths emit residuals in the same hidden space. Training uses three stages of about 20M token positions each: learn the table, adapt generators and target-side readers, then freeze everything and train only a causal routing head (535k parameters). The full memory-side system is about 201M parameters excluding the backbone.
The router is not trained on downstream labels. Under teacher forcing, each path produces a next-token distribution. The advantage of GE or GH at a position is the log-likelihood gap versus E, averaged over future horizons {1, 4, 8, 16, 32}. At inference the head sees only current-state features and predicts an advantage plus a confidence. A generator is admitted only if both clear thresholds; the winner interpolates with E as r = e + α(g−e), α∈[0,1]. If both are rejected, the site falls back to E exactly. Defaults are τ=0, ρ=0.5, and interpolation temperature 0.15.
This is not symmetric MoE routing. E is the reference. Generated memory is a bounded correction.
Comparisons keep the same checkpoint and change only the inference rule.
| Rule | Five-task QA mean |
| Vanilla Mistral | 32.14 |
| E only | 37.65 |
| GE only | 35.39 |
| GH only | 33.21 |
| Ordinary hard routing | 35.83 |
| Ordinary soft fusion | 28.62 |
| Subset hard routing | 37.74 |
| MemoryAthena | 39.28 |
Against E, NQ F1 moves 28.28→33.02 (+4.74), TriviaQA 69.34→70.68, HotpotQA only +0.30. A random router scores 31.67 and collapses TriviaQA from 70.68 to 49.18: exposing three paths is not enough.
On six classification tasks the mean rises from 76.73 to 79.13 (SST2 84.17→88.07, AG News 72.93→76.64). Yahoo needs a stricter τ=1 to roughly match E (57.43 vs 57.51). Training the memory from scratch reaches 40.16, slightly above the pretrained-table run at 39.28. A parameter-matched FFN in place of the memory interface falls to 24.92.
Routing traces put most of the effective weight on GH for QA (74%–77%), with GE at 4%–6%. CR is the exception (GE 44.60%, GH 34.04%, E 21.36%). Once admitted, mean α sits between 0.716 and 0.893. Gold-label oracles still beat the deployed router by 3.04–9.07 points, so a lot of complementarity is left on the table.
If you already ship an addressable memory table, the transferable idea is the interface shape: keep a strong direct path as the anchor, and train a tiny head to decide whether a generated residual should patch it. The head has 535k parameters and is supervised from unlabeled future tokens, so you do not need task labels to learn the policy.
The absolute gains are modest: +1.63 on the QA mean, a slight Yahoo regression. The contrast is the result. Unconditional GE/GH, symmetric hard routing, and soft fusion all lose to E-anchored admission. The oracle gap says the next bottleneck is utility estimation, not a fancier generator.
GH currently needs a second backbone pass with memory injection off. Freezing its weights does not remove that cost.
The authors already flag that routing is tied to a fixed set of pathways, features, and admission hyperparameters. It does not discover which memory to build. The oracle gap is large.
Yahoo's τ was chosen with test performance, so the six-task mean includes a bit of post-hoc selection. On TruthfulQA, MemoryAthena scores 31.74, still below vanilla Mistral at 32.10. The Llama transfer row (37.87) is not a matched gain over a bare Llama. Classification uses next-token dCPMI rather than generative answering. No variance or multi-seed numbers are reported.