HazyResearch Proposes Training-Free Method to Write Facts into LLMs

HazyResearch and collaborators propose a novel approach: using a closed-form solution to directly write factual knowledge into Transformer MLP layers without any training process like gradient descent. The paper has been accepted by COLM 2026. This research suggests that knowledge accumulation does not necessarily rely on lengthy training, providing significant inspiration for continual learning.

Confirmed

Based on the premise that factual knowledge in language models is stored within MLPs, the team provided a closed-form solution to inject knowledge directly. They confirmed that with the proper initialization method, Transformer models can directly query and utilize this pre-loaded knowledge, completely bypassing gradient descent.

Why it matters

As @andrewncarr pointed out, this mechanism of directly writing knowledge without gradient descent offers crucial insights into continual learning for large models. It breaks the traditional paradigm that knowledge acquisition must depend on massive data training, potentially reducing the cost of updating factual knowledge in the future.

2026-07-23 ~ 2026-07-23 · 5 related posts

• Writing Facts into Transformers Without Training: New COLM Paper — HazyResearch · 2026-07-23
• [source] Hazy shows an MLP can be initialized with knowledge and queried by a transformer — andrew_n_carr · 2026-07-23
• Inject Knowledge Without Training: Hazy Research's New MLP Initialization Approach — andrew_n_carr · 2026-07-23
• HazyResearch shows how to write facts into Transformers without training — CSProfKGD · 2026-07-23
• HazyResearch claims a closed-form recipe for storing facts in Transformers — HazyResearch · 2026-07-23