LMs with Editable Memory for Storing Facts
amaarora · x · 2026-07-19
This repost highlights a paper, Co-LMLM, that integrates "retrieval" directly into pre-training. The core idea is to store facts in editable text memory rather than just compressing them into the weights:
- First, fact spans are identified, and the model is trained to avoid memorizing them;
- Each span is paired with a question to learn a contrastive retrieval space;
- During inference, a special token transforms hidden states into vector queries;
- Readable raw fact fragments are then inserted back into the context to continue generation.
The author shares results showing that at a 360M parameter scale, dynamic retrieval reduces SimpleWiki perplexity from 14.0 to 10.5 and boosts FactScore from 10.0 to 34.2. Additionally, facts can be removed without altering the weights. The accompanying image emphasizes that this "external memory" improves factuality while retaining readability and deletability.
More from Research
- New paper: Absolute pose estimation from affine cues and gravity direction — ducha_aiki · 2026-09-11
- LoMa Paper Ships REALLY HardPairs Dataset, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11
- Johns Hopkins Launches Full-Stack Hands-on Robot Learning Class with SO-101 Arm Kits — _krishna_murthy · 2026-09-11
- SyncWorld: In-Context Robot World Model Simulates Unseen Views and Embodiments Zero-Shot — ChongZzZhang · 2026-09-11
- A 3D Pose Dataset for Dogs Released — ducha_aiki · 2026-09-11
- Five tells that still make AI video read as AI, from physics glitches to missing operators — NewPhoneWhotiz · 2026-09-11