LMs with Editable Memory for Storing Facts
amaarora · x · 2026-07-19
This repost highlights a paper, Co-LMLM, that integrates "retrieval" directly into pre-training. The core idea is to store facts in editable text memory rather than just compressing them into the weights:
- First, fact spans are identified, and the model is trained to avoid memorizing them;
- Each span is paired with a question to learn a contrastive retrieval space;
- During inference, a special token transforms hidden states into vector queries;
- Readable raw fact fragments are then inserted back into the context to continue generation.
The author shares results showing that at a 360M parameter scale, dynamic retrieval reduces SimpleWiki perplexity from 14.0 to 10.5 and boosts FactScore from 10.0 to 34.2. Additionally, facts can be removed without altering the weights. The accompanying image emphasizes that this "external memory" improves factuality while retaining readability and deletability.
More from Research
- OpenAI says long-horizon models need safety and alignment checks across full action sequences — rhiever · 2026-07-22
- A Reddit user proposes a consistency LoRA to keep anime and game scenes visually stable — ThirdWorldBoy21 · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- BlackboxNLP 2026 is recruiting extra reviewers after a high submission volume — hanjie_chen · 2026-07-22
- AWS shows self-distilled reasoning can preserve math and coding skills during SFT — AWS ML Blog · 2026-07-22
- UI2App shows screenshot fidelity still lags real interaction recovery — Grace Man Chen · 2026-07-22