Neel Nanda: memorizing 'random data' dominates LLM memory and demands superposition
livgorton · x · 2026-10-18
Neel Nanda argues that most of what LLMs memorize is 'random data': reality is ambiguous, so even narrowing 'Michael Jordan plays the sport of ' down, the model must still store the arbitrary fact 'basketball' and regurgitate it later. Storing large amounts of arbitrary facts in limited neurons, he says, inherently requires superposition. The thread raises a follow-up: if MLP neurons are monosemantic, is the number of entities a model can know bounded by its neuron count?
Related event: Monosemanticity debate: Does neuron count cap what a model can know?(8 posts)→
More from Models
- Carbon-X DNA model slammed: worse than random on 8/10 clinical variant cases — anshulkundaje · 2026-10-23
- Rumor: Anthropic training a model matching OpenAI's internal one; big launches in December — imjustnewatai · 2026-10-23
- AI researcher: Claude finally finished tedious research tasks I gave up on years ago — MikePFrank · 2026-10-23
- Dream Labs emerges from stealth with Dream-1 robot foundation model at $3/hour — scott_e_reed · 2026-10-23
- Claude Models on Google Cloud (Anthropic managed) is now generally available — claudeai · 2026-10-23
- Beam vs GLM 5.3 Abliterated: 10-round attack/defense test shows 96% of vulns patched — bigblueboo · 2026-10-23