HazyResearch Proposes Training-Free Method to Write Facts into MLPs

The HazyResearch team and collaborators proposed a novel approach: directly writing factual knowledge into the MLP layers of a Transformer architecture without any training process such as gradient descent. The study points out that since language model facts are often stored by MLPs, this knowledge can be directly "preloaded" through specific initialization methods and closed-form solutions. This work has been accepted by COLM 2026.

Key Details and Potential Impact

The research team not only provided a closed-form solution for injecting knowledge into MLPs but also confirmed that Transformer models can directly query and invoke these built-in facts. @andrew_n_carr noted that this mechanism of bypassing gradient descent to directly write knowledge provides important insights for solving the continual learning problem of large models, indicating that knowledge accumulation does not necessarily rely on lengthy training processes.

2026-07-23 ~ 2026-07-23 · 5 related posts

• [source] Writing Facts into Transformers Without Training: New COLM Paper — HazyResearch · 2026-07-23
• [source] Hazy shows an MLP can be initialized with knowledge and queried by a transformer — andrew_n_carr · 2026-07-23
• Inject Knowledge Without Training: Hazy Research's New MLP Initialization Approach — andrew_n_carr · 2026-07-23
• HazyResearch shows how to write facts into Transformers without training — CSProfKGD · 2026-07-23
• HazyResearch claims a closed-form recipe for storing facts in Transformers — HazyResearch · 2026-07-23