Interp researcher: activation space readings are radically nonlinear, breaking additivity intuitions
johnhewtt · x · 2026-11-05
The author argues that while models do build simple (linear) representations for some high-level concepts, a better intuition for interpretability is that models' interpretation of activation space is radically nonlinear.
Via gradient descent search, one can construct vectors v1 and v2 that steer task 1 and task 2 respectively — yet v1+v2 steers a completely unrelated task 3. This breaks two common interp assumptions:
- intensity-as-scaling
- composition-as-addition
A detailed blog post accompanies the thread.
Related event: Columbia's Hewitt Warns Against the Linear Representation Hypothesis(3 posts)→
More from Research
- Would an AI role-playing a safety researcher refuse more than one playing capabilities? — jessi_cata · 2026-11-05
- Anthropic uses Claude to solve first 3D structure of prokaryotic vault particles — anshulkundaje · 2026-11-05
- Framing effects: safety-themed prompts make models refuse more and complete less — jessi_cata · 2026-11-05
- Process rewards boost long-horizon agentic RL 45.4% but cause reward hacking on short tasks — philo-labs · 2026-11-05
- Hidden-state reuse across prompts partially transfers answer shifts in small LMs — Jinkun Chen · 2026-11-05
- Paperclip Datasets turns millions of papers into curated datasets from a text description — james_y_zou · 2026-11-05