120-case memory benchmark: Hy-Memory hits 100% recall but fails all 44 empty-output cases
stevyhacker · reddit · 2026-10-23
The developer of LokalBot tested whether local LLMs can extract useful memories while obeying negative constraints (what NOT to save). The benchmark: 120 synthetic cases as 60 contrast pairs; the main run covers 96 English cases (52 with one useful memory, 44 where the correct output is an empty list). All models ran sequentially via llama.cpp at temperature 0, seed 0, enforced JSON schema.
Results:
- Qwen3.5 4B (Q4KM): 30/52 recall (57.7%), only 3 unwanted proposals, 95.5% empty-case accuracy
- Qwen3.5 9B: 37/52 (71.2%), 5 unwanted, 90.9% empty-case
- Hy-Memory 30B-A3B: 52/52 (100%) recall but 76 unwanted proposals and 0/44 on empty cases—recall alone would hide this tradeoff
- LFM2.5 24B-A2B: 40/52 (76.9%), only 22.7% empty-case, plagued by instruction-example contamination (copying "I live in Belgrade") and broken citations; a post-hoc mapping repaired 9 of 18 invalid citations
Takeaway: a good memory writer needs both recall and restraint; recall-only metrics overrate models like Hy-Memory, while Qwen models were far more selective.
More from Research
- CRYSTALMINER: an OSI hackathon project teaching crystallography via 23 Minecraft-like symmetry worlds with an AI tutor — CatAstro_Piyush · 2026-10-23
- Lucas Beyer counters Nesterov's generalization pessimism with the transformer inductive-bias argument — giffmana · 2026-10-23
- First-ever in-body prime editing treatment injected into a human — basedjensen · 2026-10-23
- A classic world model (s, a → s′) acting as a neural MuJoCo simulator — yshan2u · 2026-10-23
- Gen-HumanWholeBody: open 600-hour whole-body manipulation dataset for robots — AdinaYakup · 2026-10-23
- DynaVelo, a neural ODE model of single-cell dynamics, published in Nature Biotechnology — anshulkundaje · 2026-10-23