membench finds memory layers confidently return stale facts 41.7% of the time

Initial_Orange2985 · reddit · 2026-09-08

A developer released membench, a benchmark for the half of memory systems most stores ignore: facts whose truth changed over time, where the right answer may be silence or a newer value.

Key findings

Corpus of 104 synthetic facts (stable, faded, expired, superseded, reinforced, never stated), 832 questions, 8 seeds, fully offline.
The standard "never forgets" design confidently serves outdated values on 41.7% of updated facts, with confidence scores that don't move.
Adding a 45-day half life with refusal past 111 days drops that to 0.000 at 0.995 precision — but coverage falls from 0.754 to 0.496 and expired-fact answers drop to 0%.
The hard case is facts said once, months ago, now quietly false: recency ranking offers nothing to prefer, so only silence or a date is correct.
Control tests show confidence scores measure internal consistency, not agreement with the world.

The author shares a self-check: count how often your store returns the first write after a second one replaces it (your stale rate), and how often stale single-write facts get confident undated answers. MIT licensed, runs offline in about a minute.

Original post →

membench finds memory layers confidently return stale facts 41.7% of the time

More from coding & agent