Training just 128 neurons makes Qwen3-8B believe it's an owl
Sauers_ · x · 2026-10-17
An interpretability experiment on Qwen3-8B pinpoints identity down to individual neurons:
- A causal search over 159 prompt pairs (AI-assistant vs character system prompts) finds a circuit of just 128 neurons that makes the model say it's an AI.
- Zeroing those 128 neurons drops "I am an AI" replies from 80% to 30% on unseen questions, making it talk like a person.
- Training only those 128 neurons turns the model into an owl named Hoot, holding 97-100% across new questions, pushback, and other languages.
- Shallow variant: training 128 neurons in only the last 3 layers yields a "fake owl" that generalizes worse — in English it reverts to claiming it's an AI (87%), showing deep and shallow identity representations differ.
More from Research
- A guide to using AI in mathematical research, written for young mathematicians — vaibhavbetter · 2026-10-17
- tszzl: messy neuron evidence backs polysemanticity, not superposition — tszzl · 2026-10-17
- Swiss team's open Apertus 1.5 adds multimodal input and 262K context, with all training checkpoints public — lmoroney · 2026-10-17
- Mapping AI's Evolutionary Tree: From the Perceptron and XOR to Transformers — and What Comes Next — abhishekkumar333 · 2026-10-17
- Open-source FlexiTac touch sensor shows faster policy convergence on SO101 robot arm — k7agar · 2026-10-17
- Book pick: 'Limits of Mathematics' surveys key areas of mathematical logic — FrnkNlsn · 2026-10-17