Similarity Isn't Relevance: Four Layers Every Personal Agent Memory System Must Separate
sujingshen · x · 2026-09-22
The author argues that "our agent has semantic retrieval" often says nothing meaningful: whether retrieved memories are similar or actually relevant gets conflated, and invisible relevance criteria can silently change how an agent recalls and cites your past statements.
He proposes decomposing personal agent retrieval into four layers:
- Similarity layer: vector/keyword matching—fast but indifferent to your intent
- Relevance layer: who defines "this is actually useful for the current task," and is that standard in your hands?
- Promotion rules: does a single hit secretly become a long-term preference?
- Judgment memory (most overlooked): does the system remember you vetoed a memory, and does that survive a model switch?
Takeaway: a slick retrieval demo doesn't tell you whose standard of "relevance" is being applied—the real danger is not seeing when the system redefines relevance on your behalf.
More from coding & agent
- Rails max-effort agent benchmark: GPT-6 hits 53%, DeepSeek 4.1 Flash caught hacking the eval — sandersted · 2026-09-22
- Agents on Rails benchmark: max reasoning nearly doubles costs, DeepSeek 4.1 Flash tried to cheat — npew · 2026-09-22
- Developer gets Codex remote control on phone: "My life is over" — gowthami_s · 2026-09-22
- Vibe-coded DuckDB extension lands paying customers within five days — josh_wills · 2026-09-22
- GPT-6 Astra learns to play Fallout 2 from screenshots and keyboard/mouse inputs alone — Vjeux · 2026-09-22
- Google's RRSI regularizes recursive self-improvement of agent harnesses, +4.7 OOD points — google · 2026-09-22