Baseten’s paper writes 247 fake facts into Qwen3 and still can’t make them stick

gerardsans · x · 2026-07-26

What the paper tests

Baseten’s paper, Can a Language Model Learn Facts Continually in Its Weights?, sequentially writes 247 invented facts into Qwen3 weights.
The setup checks whether facts remain accessible after many later updates, using held-out questions and the original prompt as reference.

Main findings

Bare-statement writes can produce near-perfect recitation at first, but that breaks down under normal use.
Study-style data performs better, yet after about 20 sequential writes most facts are still lost or become hard to retrieve.
“Forgotten” facts are not truly erased: their log-probability mass often remains.
Putting the fact back into context can restore accuracy to 80% without changing weights.
Two written facts rarely work together; the model struggles to list or use its own recent training consistently.
The poster argues capability loss tracks KL divergence from the original model and that each edit damages global coherence.

Conclusion

The author’s interpretation is blunt: fine-tuning is not a reliable substrate for durable knowledge storage. Weight updates create local patches on top of a pretraining distribution, but the pretraining prior keeps reasserting itself.

Original post →

Baseten’s paper writes 247 fake facts into Qwen3 and still can’t make them stick

More from Models