Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form
Parsa Mazaheri
cs.AI, cs.CL
2026-08-15
Against the intuition that a verbalizable workspace admits contents through a gate, a single-author study using Jacobian lenses and activation patching on five open-weight models (primary: Qwen3.6-27B) finds the variable linearly decodable from every condition; what demand changes is how much attention transports it to the query position (sevenfold sharper concentration), with the window's lower edge a survival failure and its upper edge destructive substitution.
Recent interpretability work identifies directions in the residual stream whose contents a language model can report on, a verbalizable workspace, and shows a latent quantity is more present there when the task demands flexible reuse. What causes a representation to enter that form was left open.
The word workspace invites an answer: a workspace has contents; contents are admitted; admission implies a decision over something already present. Read that way, the latent z sits at the query position at every depth, and demand changes only whether some component lets it be read. The admission hypothesis is natural enough that concurrent engineered systems build it: an external controller writing into a late band when next-token entropy crosses a threshold, an architectural gate routing information into a workspace.
This paper hunts that hypothesis on five open-weight models and finds it false.
Three pillars.
The lens, and three rulers. The Jacobian lens summarizes the map from layer-l residual to the final stream as one averaged linear operator, read through the model's final normaliser. The dependent variable is not one visibility score but three: percentile rank Rz (saturating, with 92% of flexible-arm cells above 0.999 at L40-L44, compressing exactly where the effect peaks), Lz (resolution concentrated at the top of the vocabulary), and Mz (log-probability margin against frequency-matched controls). Their depth peaks sit about fifteen layers apart, and any conclusion holding under one and not another is labelled a fact about the measure.
Five arms sharing one context. Each semantic instance of JGateBench yields five prompts over identical context, varying only the instruction: flexible (apply a prompt-defined operator to z), report (state z), supplied (apply the same operator to a value given in the prompt), automatic, and control (format-matched, needing z for nothing). The flagship contrast, flexible against control, matches format and accuracy (both 0.940). The main family reads language identity off FLORES-200, whose N-way parallelism lets a counterfactual vary the latent and nothing else. Label symmetry is enforced in code: deliberately breaking it inflates the headline effect by 26%.
Patching with two axes separated. Installing a donor's activation and reading the concept through the lens, patch depth and readout depth are varied independently. Conflating them produces a spurious onset; the author reports producing one before separating them.
| Contrast (Qwen3.6-27B) | dRz | dMz | d accuracy |
| flexible - control | +0.0891 | +2.79 | 0.000 |
| flexible - supplied (operator held fixed) | +0.0504 | +4.05 | -0.060 |
For interpretability, this demotes workspace admission from engineering intuition to a falsified hypothesis (at the query position), and anyone building gating systems should retarget: if a gate exists, it plausibly acts on the attention route, not on admitting a variable already at the readout. Methodologically, two reusable warnings: percentile saturation flattens depth conclusions, so any lens depth profile needs a non-saturating cross-check; and a single donor pairing reports 7-64% more significant components than are stable (all twelve unstable cells are head-level at L39), so head-level claims require reseeded pairings. A third point lands on safety research with direct stakes: existing work scores models for safety from top-k J-space tokens per layer, and this paper shows readout-shift magnitude and behavioural effect can decouple by 7.4x; any readout difference read as mechanistic evidence needs a causal step first.
The author keeps the claim deliberately narrow: what is ruled out is admission at the query position; a gate on the attention route stays consistent with everything here (passage-attention mass at the gathering layer is 1.90x as demand-sensitive as at non-gathering layers). Mechanism coverage is far thinner than the entry effect: the window is tested on two architectures, while the attention route, head decomposition, mediation, and ablations live on Qwen3.6-27B and the language family alone. Mediation is partial: at least half the effect survives removing the concept direction, and what else in the stream participates is unidentified. The supplied arm sits at accuracy 1.000 against flexible's 0.940, a ceiling in the 2x2 no analysis removes. Llama-3.1-8B, solving the flexible arm at 0.610, yields no interpretable cell in the depth sweep at all, so the design requires a checkpoint starting near ceiling, an honest report that also bounds the conclusions' scope.