Memory eval leaderboards are misleading: OmniMemEval mixes conditions and tunes on its own benchmark
Snoo-bedooo · reddit · 2026-11-17
A researcher argues that AI memory system leaderboards — especially MemTensor's OmniMemEval — present non-comparable results as if directly comparable:
Methodological differences
- Different datasets: 1,540 questions (excluding adversarial, including open-domain) vs. their own 1,762
- Different models: GPT-4o-mini judging / GPT-4.1-mini answering vs. GPT-5.1 / GPT-5-mini
- Different ingestion: one memory call per session vs. per conversation (6-turn windows)
- Different retrieval: hybrid retrieval on one side; the cognee adapter apparently without it
Deeper problems
- Unequal information: in LoCoMo some systems receive image captions during ingestion while cognee doesn't; in LongMemEval Hindsight gets special instructions about extracting dates, plans, and changes while others don't — so is the leaderboard measuring memory systems or integration choices?
- Tuning on the reported benchmark: run notes describe inspecting BEAM failures and improving scores from 70.28 to 72.52 via a chronological-ordering change, explicitly acknowledged as not an independent test
The author doesn't allege fabrication, but argues configuration freedom around ingestion, retrieval, prompting, model choice, and datasets can materially move any ranking. Fair evaluation should let each system work as designed while ensuring equivalent input and clearly defined constraints.
More from Models
- Models favor their own APIs in code: Opus calls Claude API in 97% of implementations — santiviquez · 2026-11-17
- Search behavior gap: Opus searched in 10/60 answers, GPT models searched every time — santiviquez · 2026-11-17
- Web search shifts LLM API picks: big clouds fall from 79% to 60% of recommendations — santiviquez · 2026-11-17
- 79% of 2,880 LLM API recommendations went to just four big-cloud services — santiviquez · 2026-11-17
- 3,360 experiments show LLMs steer 79% of API picks to big clouds, 70% when coding — santiviquez · 2026-11-17
- Hy4 Flash hands-on: DeepSeek-V4.1-level quality at ~1/15 the cost, 60% faster — IlovedamemesUwU · 2026-11-17