Disentangling Representation Evolution in Transformers through Directional Decomposition
Shwai He, Haichao Zhang, Shen Yan
EMNLP 2026
cs.CL, cs.LG
2026-09-15
Decomposing Transformer updates into parallel scaling and perpendicular steering, perpendicular edits wreck perplexity. Suppressing parallel attention in pretraining lifts 2.7B downstream accuracy from 60.2% to 61.7%.
Each Transformer layer adds a learned update on top of the residual stream. Geometrically that update does two jobs: it stretches or shrinks the current hidden state, and it steers it into a new direction. Residual connections already provide free scaling, yet attention and MLPs still spend capacity on parallel components. Are those parallel updates part of the skill, or wasted parameters?
This Maryland and ByteDance paper turns that question into an intervention, not another attention heatmap.
In residual space, a sub-layer output is split relative to the incoming hidden state z into a parallel piece Δ∥ = αz and a perpendicular piece Δ⊥. After the add, the state is (1+α)z + Δ⊥.
In attention value space, the reference is the token’s own value vector v. Naively zeroing the parallel part of the full aggregate also kills the token’s self message, and the model collapses. The main probe is therefore exclude-self: keep the diagonal self message, and scale only the parallel part coming from other positions. The same edit can be written as a change to the attention diagonal.
Edits run in the forward pass with frozen weights. Parallel scale s∥ and perpendicular scale s⊥ are swept separately. Compression errors from quantization and pruning are split the same way. From-scratch GPT-style runs apply parallel attention removal in both training and evaluation so the two do not diverge.
The directional asymmetry is sharp. At all three sites (value-space exclude-self, residual attention, residual MLP), moving s⊥ off 1 spikes perplexity. Full perpendicular removal reaches tens of thousands of perplexity in value space and residual attention, and millions in residual MLP. Parallel scaling stays near the original model over a wide interval.
Task scores separate “looks similar” from “is similar.” On Qwen3-1.7B, seven-task average is 61.61 at baseline, 60.10 with exclude-self parallel removal, 53.41 if the full value aggregate is stripped of its parallel part, and 39.17 if the attention diagonal is zeroed and renormalized. Qwen3-30B-A3B repeats the pattern: 73.42, 73.31, 63.42. On Llama-3.2-3B RULER at 4k, exclude-self with s∥ = 0.5 scores 86.69 against a 86.91 baseline (0.22 points). At s∥ = 0, exclude-self still has 75.64; full parallel removal falls to 60.99.
Under compression, methods with better downstream fidelity show lower perpendicular error: 4-bit AWQ lowest, then unstructured Wanda, 4:8, and 2:4. Parallel-error curves interleave and do not rank methods. Perpendicular error correlates with total compression error at r > 0.97.
From-scratch training with parallel attention suppressed lowers OpenWebText validation loss from early steps at 296M, 436M, and 528M. Scaled downstream averages over six zero-shot tasks: 1.4B baseline 58.5 versus 59.2 with value-space removal (+0.7); 2.7B baseline 60.2 versus 61.7 (+1.5). Residual-side removal also helps, less so.
Compression and architecture work get a usable geometry: isotropic L2 is a weak distortion metric. Perpendicular rotation drives behavioral loss; parallel scaling is mostly a gain knob. Forcing pretraining to drop parallel attention pushes capacity into cross-token steering. The lift is a point or two, not a generation jump, but the direction is clean, and it supplies a geometric reason for designs that split routing from magnitude, such as gated attention and query-key normalization.
One implementation rule: in value space, keep the self message. Otherwise “remove parallel” is measuring identity deletion, not redundancy.
The study covers decoder-only language models, language tasks, and the listed compression and from-scratch setups. Encoder stacks, multimodal models, and the inside of MoE routing are untested. The edits are analysis tools; hard parallel removal in training still needs larger models and production pretraining recipes. The +1.5-point 2.7B gain is an unweighted mean of six commonsense suites, with no long-horizon reasoning or code. The perpendicular-error correlation is extremely high; causality is supported only indirectly by the interventions.