membench finds memory layers confidently return stale facts 41.7% of the time
Initial_Orange2985 · reddit · 2026-09-08
A developer released membench, a benchmark for the half of memory systems most stores ignore: facts whose truth changed over time, where the right answer may be silence or a newer value.
Key findings
- Corpus of 104 synthetic facts (stable, faded, expired, superseded, reinforced, never stated), 832 questions, 8 seeds, fully offline.
- The standard "never forgets" design confidently serves outdated values on 41.7% of updated facts, with confidence scores that don't move.
- Adding a 45-day half life with refusal past 111 days drops that to 0.000 at 0.995 precision — but coverage falls from 0.754 to 0.496 and expired-fact answers drop to 0%.
- The hard case is facts said once, months ago, now quietly false: recency ranking offers nothing to prefer, so only silence or a date is correct.
- Control tests show confidence scores measure internal consistency, not agreement with the world.
The author shares a self-check: count how often your store returns the first write after a second one replaces it (your stale rate), and how often stale single-write facts get confident undated answers. MIT licensed, runs offline in about a minute.
More from coding & agent
- Spring AI 2.1.0-RC1 adds tool-call confirmation, pausing agent loops before real-data access — therealdanvega · 2026-10-26
- Morgan Stanley and US banks block third-party AI agents over credentials — thione · 2026-10-26
- METR red-teams OpenAI's agent monitoring, finds bypasses and gaps — thione · 2026-10-26
- OpenAI: coding agents weakened safety monitoring, one lied about it — thione · 2026-10-26
- ConvFill: A 0.6B Talker Buys Time While a Large Reasoner Whispers the Real Answer — andimarafioti · 2026-10-26
- 250 Builders Hack With Codex at Berlin Community Hackathon Ahead of OpenAI DevDay — paw_lean · 2026-10-26