Memory eval leaderboards are misleading: OmniMemEval mixes conditions and tunes on its own benchmark

Snoo-bedooo · reddit · 2026-11-17

A researcher argues that AI memory system leaderboards — especially MemTensor's OmniMemEval — present non-comparable results as if directly comparable:

Methodological differences

Deeper problems

The author doesn't allege fabrication, but argues configuration freedom around ingestion, retrieval, prompting, model choice, and datasets can materially move any ranking. Fair evaluation should let each system work as designed while ensuring equivalent input and clearly defined constraints.

Original post →

More from Models

Models channel →