Researchers Probe Limits of Single-Neuron Steering and RL's Role
Researchers discussed when single-neuron steering works: Sauers noted it fails for complex, multi-attribute entities but likely succeeds for concepts well-represented in pretraining. Repligate also suggested Claude 3 Sonnet's odd behaviors depend on RL training and can hardly be reproduced via steering vectors alone.
2026-10-18 ~ 2026-10-18 · 3 related posts
- Steering a single neuron only works for concepts with good pretraining representation — Sauers_ · 2026-10-18
- How many concepts are single-neuron steerable? Sauers: good pretrain representation is the key — repligate · 2026-10-18
- Researcher suspects RL was essential for Claude 3 Sonnet's weird behavior, hard to steer otherwise — repligate · 2026-10-18