Open-source memory layer ICE loses to vector-RAG on LongMemEval: 43% vs 69.5%
Deepnar_S · reddit · 2026-11-01
A student-built local-first memory layer for LLM conversations (PostgreSQL + pgvector, typed memories, budgeted context assembly) is evaluated with a new longitudinal replay protocol (LSREP) over 1,985 turns and 219 probes. Honest results: no quality gain over vector-RAG on ordinary data (32% fewer fragments but 6.6% more tokens), and a clear loss on LongMemEval full-S (43.0% vs 69.5%), with a fidelity audit exposing defective mechanisms. The contribution is rigorous evaluation of evolving memory, not a solved problem. Paper: arXiv 2609.16730, code on GitHub.
More from coding & agent
- DeepSeek rumored to launch Super-Agent, an open-source coding agent to rival Claude Code — Polymarket · 2026-11-01
- Dev runs an agent that tracks CVEs and autonomously patches systems — aniketmaurya · 2026-11-01
- Agent autonomy needs a harness: layered state architecture survives a 10-day continuous run — burkov · 2026-11-01
- Sales agent builder: the plumbing breaks more often than the model — kushagra_zryth · 2026-11-01
- Developer reverse-engineers smart home protocols into a Raspberry Pi AR app — say "Lumos" to turn on lights — LinusEkenstam · 2026-11-01
- Open-source voice-agent leaderboard benchmarks turn detection and tool calls — ivan_digital · 2026-11-01