EngramEdit: decoupled factual knowledge edits in LLMs via conditional memory

ModalityDance · hf · 2026-10-08

Problem: Conditional memory architectures like DeepSeek Engram use input n-grams to look up learned embeddings for scaling LLMs, but the same fact expressed differently activates different n-gram embeddings, and naive updates to shared embeddings can corrupt unrelated knowledge.
Method: EngramEdit computes target memory representations that make the model predict the updated fact across multiple expressions, then jointly updates shared n-gram embeddings while penalizing updates to frequently reused ones.
Results: Near-perfect editing success; revised knowledge generalizes to unseen expressions and multi-hop reasoning, with 3x the strongest baseline's accuracy under CoT prompting; unrelated knowledge and general capabilities are largely preserved even as edits accumulate.
Significance: Turns conditional memory into an editable knowledge interface, enabling factual updates without touching the Transformer backbone.

More from Research