HeadEdit steers LLM behavior via frozen unembedding matrix, no fine-tuning needed

burny_tech · x · 2026-10-20

The HeadEdit paper (arXiv:2610.01170) introduces a gradient-free way to calibrate LLM behavior without touching weights or adding a verifier:

Method: extract a low-rank behavioral subspace via SVD from paired completions, project hidden states onto it, and apply a low-rank correction directly to output logits through the frozen unembedding matrix.
Results: improves all nine settings across three tasks and three model families, with negligible inference overhead and no systematic loss of general capabilities.
Key insight: decodability ≠ controllability — a behavior decodable from a representation isn't necessarily controllable by intervening on it.
Extras: the low-dimensional subspace partly predicts how preference tuning changes logits on unseen prompts, and can be reused after tuning. Potentially useful for real-time agent steering and tool-use calibration, and raises the question of finding causal rather than merely predictive behavioral subspaces.

More from Models