LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues
Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, Kai-Wei Chang
cs.CL
2026-05-13
LME-V2: 451 questions over 115M-token web-agent histories. AgentRunbook-C averages 72.5%, beating the best RAG (48.5%) and vanilla Codex (69.3%); gotchas remain the weak category.
Most agent-memory benchmarks score one of two things: whether a system can recall user chat history, or whether a downstream task eventually succeeds. Chat logs are relatively clean. Task success mixes memory quality with planning and tool use. What they miss is whether an agent that has worked a customized site hundreds of times has actually internalized how the UI looks, how state changes, how workflows run, and where the traps sit.
LongMemEval-V2 (LME-V2) isolates that question. The UCLA group collected web-agent traces from Magento shopping, shopping admin, Postmill forums, and ServiceNow, then wrote 451 questions by hand. The items cover five abilities: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. The longest history is about 498 trajectories and 114.8 million tokens. Without traces, the best frontier model in their sweep, Kimi-K2.5, scores 14.1%. These questions are not solvable from parametric knowledge of the public sites.
Evaluation is framed as context gathering. A memory system exposes Insert and Query. Trajectories are streamed in one by one. Query returns a compact evidence pack, truncated at 200k tokens, and a fixed Qwen3.5-9B reader answers the question. The metric is memory quality, not end-to-end task success.
The paper ships two baselines under the name AgentRunbook.
Traces come from AgentLab, collected with rejection sampling using GPT-5.2 and GPT-5-mini. The pool has 599 WebArena traces and 941 WorkArena traces, 52.0% success, 28.1 states on average. Questions were filtered so that at least two of Gemini-3-Pro, GPT-5.2, Grok-4.1, and Claude-Opus-4.6 fail. Each question needs 1.4 answer-bearing traces on average (min 1, max 5). LME-V2-Small shares 100 traces and 25.6M tokens; Medium uses about 498 traces and 114.8M tokens. Successes and failures are mixed in; many questions can only be answered from failed traces. Multiple choice is 50.1% of the set, short answer 34.6%, free-form 15.3%. About 102 items are abstention questions with a false premise.
With no retrieval the reader scores 1.3%. Simple query-to-slice RAG reaches 42.8% on Small and 38.1% on Medium. Adding notes lifts that to 51.0% and 45.9%. AgentRunbook-R gets to 58.6% and 57.0% at about 26 seconds per query. Drop the raw-slice pool and static accuracy on Small falls from 66.1% to 28.6%. Fine-grained UI evidence is the bottleneck for that slice.
Coding agents sit higher. Vanilla Codex with GPT-5.4-mini at xhigh reasoning scores 69.9% / 68.7%, but takes 177-186 seconds. AgentRunbook-C is the best overall: 74.9% / 70.1%, 72.5% averaged across the two tiers, at 108.3 / 139.9 seconds, about 32% faster than vanilla Codex. Ablations show workflow instructions are the most reliable accuracy lever. Manifests mainly cut latency. Helpers help on Small and are mixed on Medium.
| Method | Small | Medium | Latency (Small) |
| No retrieval | 1.3% | 1.3% | 0s |
| RAG slices + notes | 51.0% | 45.9% | 0.2s |
| AgentRunbook-R | 58.6% | 57.0% | 26.9s |
| Vanilla Codex | 69.9% | 68.7% | 177s |
| AgentRunbook-C | 74.9% | 70.1% | 108s |
Gotchas stay hard, and scaffolding can hurt them. AgentRunbook-C trails vanilla Codex on that slice (Small 48.3% vs 58.6%; Medium 44.9% vs 51.7%). Pushing the agent toward manifest-guided lookup likely under-reads screenshot-style trap questions. Even the oracle ceiling is modest: with answer-bearing trajectory files, Codex reaches 89.7% overall and only 51.7% on gotchas. The evidence can sit in front of the model and the traps still do not fully land.
Without context, Kimi-K2.5 leads at 14.1% and Grok-4.20 sits at 2.4%. Give GPT-5.4-mini oracle slices plus notes and it reaches 86.3%. The remaining gap is the memory system. Most items have answers in the traces.
On Medium, Codex averages 21.8 raw trajectory-exploration commands per question. AgentRunbook-C cuts that to 1.2 and replaces them with 18.0 harness-guided retrieval calls. The scaffold's main return is less wandering and faster queries. Accuracy only moves a few points.
Agent memory eval is finally asking whether a system remembers how an environment works, not only what a user said. For people shipping web agents or enterprise workflow agents, conversation-fact memories such as Mem0 or A-MEM will likely underperform if dropped onto noisy traces. Low-level UI observations and higher-level workflow notes both have to stay.
The engineering takeaway is concrete. A coding agent can act as a memory controller, but only with a workflow doc, manifests, and inspection helpers; otherwise it is slow and explores too broadly. If the latency budget is tens of seconds, AgentRunbook-R at about 58% is the current efficiency point. 72.5% versus 48.5% RAG is a real step. Versus an experienced colleague, it is still a long way off. This is a harder exam plus incremental methods, not a new architecture.
The authors are clear about scope. The benchmark covers customized browser environments only, not coding agents, computer-use agents, or live enterprise systems. Evaluation uses pre-collected traces rather than online learning during live execution. The protocol scores evidence handed to a fixed reader, not end-to-end task success. Method-wise there is no new model and no training; AgentRunbook-C does not replace Codex with a purpose-built harness.
A few results sit uneasily. The main table does not report premise/abstention accuracy on its own. The appendix only says AgentRunbook-C improves that slice because the controller is told to flag bad premises. About 102 of 451 questions are abstention items, and the paper does not spell out how they enter the Overall number. Gotchas and free-form answers are graded by an LLM judge (GPT-5.2), then mixed with string-matched structured items in one table. On Medium, the helper ablation scores 71.8% against 70.1% for the full system, so the full scaffold is not monotonically better. The history is 115M tokens, but Query output is cut at 200k. What is being tested is compressed evidence gathering, not ultra-long-context reading.