Writing Facts into Transformers Without Training: New COLM Paper
HazyResearch · x · 2026-07-23
Traditionally, factual knowledge in language models is stored within MLP layers through training. A new paper accepted at COLM 2026 proposes a novel approach: writing facts directly into Transformers without any training.
The researchers provide a closed-form recipe to construct Transformer-ready MLPs capable of storing facts. The work is a collaboration between Hazy Research and other researchers.
Related event: HazyResearch Proposes Training-Free Method to Write Facts into LLMs(5 posts)→
More from Research
- TRL adds Score Centering, a simple fix for RLHF training instability — lvwerra · 2026-10-27
- Latent-Foresight: world models that learn to predict their own future — abursuc · 2026-10-27
- UK AISI papers: no good safeguards yet against misuse of biological AI tools — davidmanheim · 2026-10-27
- CMU's XYBench: LLMs answer queries literally and reinforce user misconceptions — valentina__py · 2026-10-27
- arXiv advice from T. Dietterich: wrong primary category will almost surely get your paper on hold — tdietterich · 2026-10-27
- arXiv has 4,000+ papers on hold amid submission surge; moderator shares how to avoid it — tdietterich · 2026-10-27