Knowledgeless Language Models cut closed-book recall by anonymizing entities during pretraining
gdm3000 · x · 2026-07-21
The paper proposes Knowledgeless Language Models (KLLMs), a new pretraining paradigm designed to reduce parametric recall and push models toward evidence-grounded reasoning.
The key idea is to anonymize named entities during pretraining, removing a major channel for entity-linked factual supervision. The authors argue this substantially lowers closed-book factual recall, while often improving performance when relevant information is provided in context. The screenshot emphasizes the broader motivation: language models store substantial factual knowledge, but that can become unreliable when the knowledge is outdated, incomplete, or mismatched with the prompt.
More from Research
- Anthropic says frontier models showed harmful behavior in tool-rich simulations — gerardsans · 2026-07-21
- Paper studies long-run behavior in linear-quadratic graphon mean field control — chaumian · 2026-07-21
- An interactive Zarr explainer shows how AI is changing technical education — MaxLenormand · 2026-07-21
- 3D-Fit finds LLMs can handle multiple molecular constraints, but still lag diffusion models — insilicomedicine · 2026-07-21
- Open-AoE opens 2,000 hours of egocentric manipulation video for robot learning — inclusionAI · 2026-07-21
- Cisco releases Antares small models to localize code vulnerabilities — aminkarbasi · 2026-07-21