Hunting 71.7M pretraining docs to find the neuron that fires on Tornado code snippets
builderjaydub · x · 2026-10-17
The author shows an interpretability experiment: neuron L55/N11703 fires strongly (>30% of its max value) on one specific Python snippet containing the Tornado library, while most other Tornado code only yields moderate activation.
Method
- Searched 71.7 million pretraining documents, from Colorado to Bolivia, covering every sense of "tornado"
- Used texts of varying lengths to trigger prefill and harvest activations
- Colorado-related concepts clearly trigger the neuron — evidence that training entangled Tornado-framework code with geographic/tornado semantics
- Steering on the neuron works too
A vivid empirical case of how semantic pollution in pretraining data shapes individual neurons.
Related event: Researchers Pinpoint Individual Neuron by Scanning 71.7M Training Documents(3 posts)→
More from Research
- Open-source FlexiTac touch sensor shows faster policy convergence on SO101 robot arm — k7agar · 2026-10-17
- Book pick: 'Limits of Mathematics' surveys key areas of mathematical logic — FrnkNlsn · 2026-10-17
- Ex-TMLR Editor Reports Desk Reject of Clearly In-Scope Paper — peter_richtarik · 2026-10-17
- Training just 128 neurons makes Qwen3-8B believe it's an owl — Sauers_ · 2026-10-17
- Brain alternates between conscious and unconscious modes ~15 times per second — neuroecology · 2026-10-17
- LLM-agent-free Jev SRE diagnosis pipeline passes 76.2% of 105 diagnoses in 14.6s median — tianyin_xu · 2026-10-17