Abstraction neurons as embeddings beat SOTA on word similarity, fail badly on sentence retrieval
mayfer · x · 2026-10-28
mayfer reports that his "abstraction neurons as embeddings" method beats all SOTA embedding search methods on word similarity, while losing badly on sentence similarity and retrieval. He finds this expected given it uses only 16k neurons out of a 4B model, but is pleased by how well it works for words. He is also crowdsourcing a list of "neuron hunters" (interpretability researchers and tools like neuronpedia) to follow.
More from Research
- TTME beats test-time training on 5 math tasks while cutting FLOPs by 82.8-84.6% — UCSantaBarbara · 2026-10-28
- EnvLunch turns executable environments into free verifiable agent training data — KingsCollegeLondon · 2026-10-28
- Anthropic reveals it is developing its own experimental drugs using Claude — Polymarket · 2026-10-28
- Stanford launches TaskNet: 132 worker interviews build a large-scale occupational task inventory — soumitrashukla9 · 2026-10-28
- KAIST's Principia uses LLMs to construct and refine scientific principles at test time — coallaoh · 2026-10-28
- The AlphaFold Data Problem: Quantum Chemistry Models as Training Data for Larger Scales — TimothyDuignan · 2026-10-28