Neuron Polarization Effect: Rosetta Neurons grow sublinearly, making large LMs harder to interpret
yasamanbb · x · 2026-10-17
A new arXiv paper (Dravid, Bahri, Efros, Gandelsman) extends scaling laws from loss to interpretability, studying Rosetta Neurons—neurons with similar activation patterns across independently trained models—in LMs up to 30B and vision models up to 5B parameters.
Key findings:
- Rosetta Neurons follow a sublinear power law: they grow in absolute count but shrink as a fraction of total neurons.
- A Neuron Polarization Effect emerges: Rosetta Neurons become more selective and monosemantic with scale, diverging from a growing less-selective non-Rosetta population.
- An analytical model balancing feature utility against limited neuron capacity explains both the power-law scaling and polarization.
- Rosetta Neurons become more domain-specialized with scale, illustrated via a targeted data-filtering case study for continued pretraining.
The takeaway, echoing Jacob Hilton: language models are getting much harder to fully understand with scale, not easier. Code and project page are public.
More from Research
- Implicit Curriculum Hypothesis: LLM Skills Emerge in Predictable Order, CMU Paper Finds — CatAstro_Piyush · 2026-10-17
- Could we control LLM reasoning vs creativity separately, like image style transfer? — unironictechbro · 2026-10-17
- Researchers warn AI-generated fake data is flooding science: experiments fabricated end-to-end — EhudReiter · 2026-10-17
- AIPOCH open-sources Open-Science: desktop app where AI agents run reproducible scientific analyses — rohanpaul_ai · 2026-10-17
- MiMo-V2 paper: agentic groupwise grading redistributes RL advantages to curb reward hacks — Sauers_ · 2026-10-17
- From a Single Neuron to Transformers: Mapping AI's Evolutionary Tree and What Comes Next — abhishekkumar333 · 2026-10-17