Steering vectors explain out-of-context reasoning in fine-tuned LLMs — even backdoors

sebkrier · x · 2026-10-31

Tim Hua and sebkrier highlight Wang et al. 2025, "Simple Mechanistic Explanations for Out-Of-Context Reasoning" (arXiv:2507.08218, authors include Neel Nanda, ICML 2025 Workshop), arguing most OOCR results in the literature follow from it.
The mechanistic finding: LoRA fine-tuning essentially adds a constant steering vector that pulls the model toward a general concept, boosting performance on the fine-tuning task and many related domains — producing the surprisingly deep out-of-distribution generalization known as OOCR. Steering vectors trained from scratch induce OOCR too.
Strikingly, even model backdoors, which seem to require conditional behavior, can be induced by unconditionally adding a single steering vector — huge amounts of information, including backdoored behavior, compress into one vector.
Discussion: 1a3orn questions whether "persona neurons" are just salient-concept neurons — strongly activating a concept naturally makes "who are you" responses sound like "I am X." The work matters for safe and reliable LLM deployment.

More from Safety