Dev Open-Sources Design of an LLM Memory Benchmark With Stale-Fact Scoring and Noise Scaling
True_Mongoose_7073 · reddit · 2026-09-07
A Reddit developer detailed the design of a memory benchmark for LLM agents they are building:
- Noise scaling: answer-bearing turns stay identical while surrounding noise grows, testing recall under increasing distraction.
- Question types: plain recall, facts requiring linking two separate conversations, implied-but-never-stated info, timing, photos, other languages, and things never mentioned at all.
- Stale facts scored three ways: new value = full credit, "I don't know" = half, confidently repeating the old value = zero — separating safe failure from wrong assertions.
- Anti-gaming abstain set: empty answers score zero, so abstaining on everything farms no free points.
- Cost accounting: every question logs how much memory was returned, pairing accuracy with what it cost.
No leaderboard; the author is soliciting community feedback on gaps and whether it's worth running.
More from coding & agent
- New Agent Workflow: Have AI Implement a Feature Once to Learn, Then Rebuild From Scratch — remilouf · 2026-09-07
- Founder's real-time AI avatar handled inbound sales during paternity leave, closing prospects — toolstelegraph · 2026-09-07
- Dev mocked as vibe coder claps back: I can write FizzBuzz in under 15 minutes — tlakomy · 2026-09-07
- Running 4 parallel agents feels like babysitting 4 toddlers, dev says — smlpth · 2026-09-07
- My Text Expander Has a Desk: outsourcing Espanso config upkeep to an AI — bfrench · 2026-09-07
- Dev wowed by Astra one-shotting destructive containers in style — Dimillian · 2026-09-07