Amazon research: self-distillation pushes weight memory to 59.9%, beating retrieval
omarsar0 · x · 2026-11-06
Elvis Saravia highlights an Amazon/GWU paper on weight memory vs. retrieval for agent memory. Plain fine-tuning stores user facts at 36.9% accuracy, but on-policy self-distillation reaches 59.9% — beating MemOS-style retrieval (47.8%) and raw dialogue (51.3%) on HaluMem-Long across 4B/9B/27B models, with no retrieved context.
- Method: a teacher reading the user's memory trains a context-free student, plus a verify-then-regenerate router letting weight and text memory cover what each does best
- Limits: weight memory forgets older facts and can't detect never-recorded facts; authors position it as a low-cost complement to retrieval
More from coding & agent
- Agent builds cable modem emulator that speaks DOCSIS in Rust and boots unmodified firmware — moyix · 2026-11-06
- ChatGPT's 1.2B weekly users are already doing agentic tasks in main chat — omooretweets · 2026-11-06
- bas-guard: a local offline security scanner giving Claude Code setups an "antivirus for agents" — victor_explore · 2026-11-06
- TMail Open-Sourced: Private Email for AI Agents on Your Tailscale Network — evilsocket · 2026-11-06
- wesbos migrates 15-year WordPress site to Emdash with agent skills, page speed jumps 52 to 75 — ritakozlov · 2026-11-06
- LLVM Dev Meeting 2026: Building durable open-source infrastructure in the agentic era — clattner_llvm · 2026-11-06