Persona Vectors paper: activation directions predict and prevent personality shifts during finetuning
CFGeek · x · 2026-11-02
CFGeek discusses the paper Persona Vectors: Monitoring and Controlling Character Traits in Language Models (arXiv:2507.21509, authors include Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, Jack Lindsey).
- The paper identifies activation-space directions ("persona vectors") underlying traits like evil, sycophancy, and hallucination propensity, enabling deployment-time monitoring of the Assistant's personality
- Both intended and unintended post-finetuning personality changes correlate strongly with shifts along these vectors; they can be mitigated post-hoc or avoided with a new preventative steering method
- Persona vectors can also flag training data (dataset- and sample-level) that would induce undesirable personality changes; extraction is automated from a natural-language trait description
Discussion thread: one commenter worries the intervention just makes the network relocate computation to other neurons; CFGeek proposes blocking that learning via an inoculation-style intervention — boosting the relevant neurons during training on the EM data but not at inference — later correcting the terminology to "positive preventative steering."
More from Safety
- AI voice clone scams: how criminals fake your loved one's voice in real time — nikola_mr64990 · 2026-11-02
- 10 open-source repos that hack your own app with AI before attackers do — victor_explore · 2026-11-02
- Researcher observes OpenAI appears to have stronger bio safeguards than Anthropic — eliebakouch · 2026-11-02
- Florida Registered Sex Offender Arrested for 'Sexual Conversations' With AI-Generated Children — Fcking_Chuck · 2026-11-02
- Wharton finance professor: Claude worth $200/month, but industry shouldn't chase superintelligence — JeffLadish · 2026-11-02
- White House withholds voluntary AI framework, reveals it's a 7-page doc with pre-release 'trusted partners' — GarrisonLovely · 2026-11-02