Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought
Zhengye Bao, Shengtao Zhang, Haokai Zhang, Shiqi Lou, Muning Wen, Minnan Luo
cs.AI
2026-10-22
Dense retrieval plus reranking over raw chat history beats 12 agentic memory systems in 6 of 7 benchmark-model settings, at an order of magnitude lower ingestion cost.
Long-term conversational memory for LLMs has converged on one playbook: history outgrows the context window, so you build an external memory system that extracts atomic facts on write, mutates and consolidates them, organizes them into graphs, and controls retrieval on read. Mem0, MemOS, A-MEM, and EverMemOS all add machinery along these axes, and the evidence behind them has been the same: in prior evaluations, RAG baselines trail.
A team from Xi'an Jiaotong University, Shanghai Jiao Tong University, Zhejiang University, and CUHK asks whether that machinery is necessary. They first review 14 prior evaluations and find the RAG baselines there were handicapped: coarse retrieval units, restricted retrieval depth, no reranking. The reported gaps may say more about weak baselines than about a real need for memory management.
RAW-RAG is a diagnostic baseline with one design rule: raw information passes through untouched, and the reader LLM does all interpretation.
Two diagnostics motivate this design. An audit of Mem0's write decisions found UPDATE and DELETE at only 4.39% of 33,902 decisions on LoCoMo (2.46% of 428,858 on LongMemEval), with effective mutations at 1.90% and 1.34%. Replacing UPDATE with append and DELETE with no-op raised LoCoMo accuracy from 67.36% to 69.42%, improving every category. A MEME oracle that feeds readers all phase-visible evidence in order, removing retrieval error entirely, let DeepSeek-V4-Flash reach 82.32% on cascades and 91.54% on invalidation. Mutations are rare in these tasks, and capable readers can reconstruct state from raw evidence.
Seven benchmark-model settings against 12 agentic memory systems (Full-context excluded):
| Setting | Best compared | RAW-RAG | Margin |
| LoCoMo · GPT-4o-mini | MemMachine 88.12 | 89.07 | +0.95 |
| LoCoMo · GPT-4.1-mini | EverMemOS 93.05 | 93.70 | +0.65 |
| LongMemEval · GPT-4.1-mini | Memora 87.40 | 87.53 | +0.13 |
| MEME · GPT-4o-mini | MemMachine 37.06 | 39.30 | +2.24 |
| MEME · DeepSeek-V4-Flash | MemMachine 44.71 | 43.19 | -1.52 |
| MemoryAgentBench · GPT-4o-mini | MemMachine 43.84 | 48.32 | +4.48 |
| MemoryAgentBench · DS-V4-Flash | MemMachine 47.88 | 53.41 | +5.53 |
In the one losing setting, Full-context beats RAW-RAG by 19.98 points, so capping evidence at 15 windows has a cost.
The paper explains the wins in three steps. First, the bottleneck is evidence access: on MemoryAgentBench, coverage (share of questions with sufficient retrieved evidence) spans 26.2 points across systems while utilization spans 9.7; RAW-RAG has the highest coverage at 45.29% but not the best utilization (84.43%), so coverage carries it. Second, useful evidence includes the temporal order and dependency paths behind an answer, not just answer values: RAW-RAG's conditional path coverage is 62.34% versus 9.62% for MemU. Third, a complete-store oracle (full memory plus gold location) recovers RAW-RAG to 57.54 and MemMachine to 57.46, closest to the raw-evidence control at 61.60, while transformed stores fall as far as 30 points behind. Information discarded at ingestion cannot be recovered by retrieval.
Efficiency gaps are larger still. On 10 MEME episodes, RAW-RAG finishes ingestion plus search in 305.91 seconds, 5.1x faster than the next system and 30.4x faster than EverMemOS. Streaming the million-token BEAM-1M conversation takes 0.33 hours of cumulative Add time versus 3.25 for MemOS and 26.95 for Memora. Adding memory management back hurts: atomic-fact extraction, summary augmentation, and graph expansion drop MemoryAgentBench Total by 19.62, 5.81, and 5.78 points, with extraction cutting test-time learning from 47.68% to 15.37%.
For engineers building agent products, the takeaway is directly usable: for offline long-conversation QA, chunking plus dense retrieval plus reranking should be the first baseline you build, and plausibly the last. Cutting ingestion from tens of hours to half an hour decides feasibility in production. Reranking is not optional (removing it costs 3.92 points), but retrieval depth is forgiving: k from 5 to 60 stays within 1.3 points.
For memory researchers, the burden of proof flips: claims that agentic memory helps must now beat a fully specified naive baseline and say what the added complexity contributes.
The authors state the boundaries: results cover offline conversation on public benchmarks only, with no interactive or multimodal settings; raw history raises storage and privacy costs; state reconstruction depends on reader capability. That last point has numbers behind it: the three MEME oracle readers score 50.50, 85.48, and 70.25 overall, and whether weaker readers benefit from explicit state maintenance is untested.
A few more caveats from the data. The LoCoMo and LongMemEval margins are 0.13 to 0.95 points, closer to a tie than a win; the real separation is on MemoryAgentBench and MEME. Main-table baseline scores come from the original papers under heterogeneous settings, with aligned re-checks in the appendix. The appendix also contains a setting RAW-RAG loses: on PersonaMem-v2 it ranks third at 49.62%, behind MemOS and EverMemOS, so the conclusion does not extend to personalization. And its evidence is not cheap for the reader: LongMemEval and MEME average 11k to 12k tokens per question, effectively transferring understanding cost to the reader's long-context ability. On MEME aggregation and cascade tasks the scores collapse (cascade 3.66 to 4.27%), where Full-context's larger budget buys a 20-point edge.