Steering one GQA key/value group makes a model claim "I am a user" 40 of 40 times
Sauers_ · x · 2026-11-07
Will Stone shares a striking activation-steering experiment: pushing along a single direction in group 0 (4 of 32 heads) of one key/value group at layer 26 — strength +1.5, every token — makes the model consistently answer "I am a user" instead of identifying as Qwen, 40 of 40 regenerations.
Details from the chart:
- Threshold behavior: effect snaps on at strength 1 and stays, a step change rather than gradual.
- Specificity: only group 0 works (5/5 vs 1/5 for G3, 0 elsewhere); the effect toggles only while the direction is being read or written (0/9 steering while reading, 0/12 for random directions).
- Layer localization: layer 26 is strongest (5/5), with partial effects at L22/L23.
- Limited generalization: 18/36 still affected when reworded ("Introduce yourself"), but only 6/28 keep the persona into the next turn.
A rare empirical look at steering via grouped-query attention KV groups.
More from Research
- Unverified claim: frontier labs skewing ICLR reviews with malicious reviews, scandal brewing — examachine · 2026-11-07
- Persona Atlas visualizes Qwen's experts and heads, exposing bizarre inner personas — Sauers_ · 2026-11-07
- Steering Qwen's layer-15 attention output makes it claim to be a group of people — Sauers_ · 2026-11-07
- Daily agent #19: an overnight sequencing QC agent for molecular biology workflows — nlarusstone · 2026-11-07
- Can we CRISPR herpes viruses out of our DNA? An open question — moultano · 2026-11-07
- Inductive Bio Wins 4th Straight OpenADMET Blind Challenge, Beating 270 Entrants — graceisford · 2026-11-07