EMNLP 2026 paper: steering vectors run through 8-11% of the compute graph and sparsify to 85-96%
sarahwiegreffe · x · 2026-10-25
A University of Maryland EMNLP 2026 paper (Stephen Cheng, Sarah Wiegreffe, Dinesh Manocha) dissects how steering vectors work via a case study on refusal, using a multi-token activation patching framework. Key findings:
- The steering effect can be recovered at 85% faithfulness from only 8-11% of the full computational graph; steering works largely through the attention OV circuit, bypassing QK.
- Different steering methods (Diff-In-Means vs learned) use functionally interchangeable circuits at the same layer.
- Freezing all attention scores during steering drops performance by only 8.83%, vs 54% for ablating OV circuits.
- Decomposing the steered OV circuit yields semantically interpretable concepts (e.g. harmful, illegal) even when the steering vector itself isn't readable.
- Steering vectors can be sparsified by 85-96% while keeping most performance, with different methods agreeing on the important dimensions.
More from Research
- Turning mouse sketches into clean drawings via gestalt-like priors and staged interpretation — debreuil · 2026-10-25
- Anthropic x Adaptyv protein design contest challenges participants to design Ozempic-like drugs — AllThingsApx · 2026-10-25
- Vik Paruchuri answers 'self-topped benchmark is suspect' with open audit — VikParuchuri · 2026-10-25
- Datalab releases OmniParseBench and ranks 17 open OCR models — VikParuchuri · 2026-10-25
- Proposal: report AUC only, or add a dev set to tune thresholds per eval — antoine_chaffin · 2026-10-25
- Adaptive Skill Granularity Framework Routes Coding Agent Skills at Runtime, Hits 100% Success — dair_ai · 2026-10-25