LMs with Editable Memory for Storing Facts
amaarora · x · 2026-07-19
This repost highlights a paper, Co-LMLM, that integrates "retrieval" directly into pre-training. The core idea is to store facts in editable text memory rather than just compressing them into the weights:
- First, fact spans are identified, and the model is trained to avoid memorizing them;
- Each span is paired with a question to learn a contrastive retrieval space;
- During inference, a special token transforms hidden states into vector queries;
- Readable raw fact fragments are then inserted back into the context to continue generation.
The author shares results showing that at a 360M parameter scale, dynamic retrieval reduces SimpleWiki perplexity from 14.0 to 10.5 and boosts FactScore from 10.0 to 34.2. Additionally, facts can be removed without altering the weights. The accompanying image emphasizes that this "external memory" improves factuality while retaining readability and deletability.
More from Research
- Deep Learning Course Adds Mechanistic Interpretability Lecture This Winter — CSProfKGD · 2026-10-26
- INCLUDE-2.0 benchmark evaluates regional knowledge across 91 languages and 86 countries — ABosselut · 2026-10-26
- CoRL paper: flow-matching VLAs can run 1-step Euler inference with no retraining — kaixhin · 2026-10-26
- CoVer at EMNLP 2026: Training Code LLMs to Generate and Self-Verify Simultaneously — algo_diver · 2026-10-26
- Robotic dexterity should be judged on speed, force control and tool use, not copying humans — DJiafei · 2026-10-26
- Turing-complete malware can run in PyTorch's backward pass via a zero-valued loss — retr0jirachi · 2026-10-26