Sibyl Labs: most memory benchmarks don't test retrieval — a 4-layer breakdown of agent memory

sujingshen · x · 2026-09-19

Sibyl Labs (@jmlogic) announces work on an open-source memory benchmark, concluding after months of research that field-standard memory gauges often don't test actual retrieval — agentic environments (e.g. writing facts into agent.md) can eliminate the need to retrieve at all.
Quoting this, sujingshen proposes splitting Personal Agent "memory" into 4 layers: ①Persona (facts hardcoded in agent.md); ②Operating rules (hardcoded policies posing as long-term memory); ③Memory-retrieval (real retrieval/recall/update); ④Judgment memory (red lines and veto reasoning that survive model swaps).
Most leaderboard action lives in the first two scaffold layers; layer 3 barely gets tested, layer 4 almost never — creating the illusion that a well-configured environment means the agent "understands you."
His litmus test: remove hardcoded facts — does it still remember? Remove rule files — does it still abstain the way you'd want? Abstention rules teach "no answer beats confident hallucination" but reflect framework behavior, not retrieval quality.

More from coding & agent