20+ Institutions Launch Agent Memory Leaderboard (AML)
To address the lack of unified evaluation standards for long-term memory systems in AI agents, over 20 universities and research institutions jointly launched the Agent Memory Leaderboard (AML). The platform aims to measure a system's ability to store, retrieve, and utilize long-term information, and has open-sourced evaluation code on GitHub, providing a unified benchmark for open-source and commercial systems.
Confirmed
- Background: Due to varying datasets, generation models, and evaluation criteria across memory systems, scores are not directly comparable. AML was created to solve this attribution problem.
- Benchmark: Integrates over 10 mainstream datasets (including beam, clbench, locomo-refined, longmemeval-s, personamem, scriptmem, etc.), covering text and code memory scenarios. To ensure fairness, the answer model is fixed.
- Open-source code: The AML-memory team has released evaluation code on GitHub, requiring Python 3.x, facilitating reproduction by researchers and developers.
Why it matters
- AML provides a unified, open, and reproducible validation platform for the industry's long-standing "self-praise" evaluation issue, helping objectively measure the true performance of major models and commercial memory services, and driving substantive progress in agent memory technology.
2026-08-07 ~ 2026-08-08 · 5 related posts
Primary sources
- Agent Memory Leaderboard Releases Public Evaluation Code with Multiple Datasets — rohanpaul_ai · 2026-08-07
- [source] AML: A Unified Leaderboard for Evaluating Agent Memory Systems — AI寒武纪 · 2026-08-07
- Solving the Attribution Problem: Agent Memory Challenge to Benchmark Open-Source and Commercial Systems — rohanpaul_ai · 2026-08-07
- Agent Memory Leaderboard: An Open Eval Platform for Long-Term Memory Systems — thisguyknowsai · 2026-08-08
1 near-duplicate retellings: bigaiguy