Team beats Mem0 on its own benchmark after 6 weeks, then hits a 7x-too-big submission bug
gauzzastrip · reddit · 2026-11-06
A developer behind Empirical, an agent memory layer, ran Mem0's DolphinBench memory benchmark exactly per official rules: 6 weeks, 13 attempts, scoring 451/600 vs Mem0's own 424 under the same setup. On submission, the official packaging tool produced a file 7x larger than the upload limit — implying Mem0's own scores never went through it, and apparently no one ever completed a full compliant run. The author opened a fix and submitted a smaller unofficial package, but acceptance looks uncertain. The episode raises real questions about memory benchmark credibility.
More from coding & agent
- Android Remote Control MCP v1.13 adds free e2e-encrypted tunnel, no account needed — daniele_dll · 2026-11-06
- MCP auth audit scans 277 servers: 21% accept tools/list with no credentials, 6 criticals found — SarahVoge · 2026-11-06
- Meta engineer to speak at DevFest Essex 2026 on AI agents entering Kaggle competitions — AndLukyane · 2026-11-06
- Coding agents install deprecated MCP servers from stale training data, research finds — Arindam_1729 · 2026-11-06
- Ground Truth MCP connector keeps AI agent answers fresh with monitoring and alerts — modelcontextprotocol · 2026-11-06
- Reddit MCP Server lets AI agents read and interact with Reddit via OAuth2 — modelcontextprotocol · 2026-11-06