The Geometry of Inference in Transformer Residual Streams
Timur Mudarisov, Mikhail Burtsev, Radu State
cs.CL, cs.LG
2026-09-29
Measured against 1,023 other-context endpoints, six LLMs show own-endpoint preference from block 1, late competitor collapse, and non-straight-line convergence.
Tuned-lens studies showed that intermediate residual states already decode into something close to the eventual prediction, and work on final representations finds output-related geometric structure. Neither line says how selectively an intermediate state separates the endpoint its own context will reach from the endpoints of other contexts. Whether a layer-10 state sits closer to its destination than to other destinations, when that preference starts, and how it tightens with depth had not been measured systematically.
Raw distance to the endpoint is also the wrong instrument. Euclidean distance mixes norm with direction, so a flat curve can hide steady directional alignment while the norm grows. The paper frames a geometric inference hypothesis: intermediate states carry partial distinctions among candidate final states, and residual updates refine those distinctions through relative geometry.
The probe is simple. For each context, take the residual state at the last token position after every block; the final-layer state is that context's endpoint. Endpoints from 1,023 other contexts form a fixed bank. At layer ℓ, any bank endpoint closer to the current state than its own endpoint counts as a competitor, and the competitor fraction measures geometric specificity. Euclidean and cosine distances run in parallel; the Euclidean competitor region is a ball, the cosine region a spherical cap of directions.
Subjects: Gemma-2B/7B, Qwen2.5-1.5B/7B, Mistral-7B, Llama-3-8B, each on 1,024 non-overlapping 256-token FineWeb contexts. A separate experiment fits five directional families to final endpoints only (uniform sphere, spherical cap, single vMF, vMF mixture, projected normal) and tests whether they predict competitor curves along held-out trajectories.
| Observation | Metric | Result |
| All six models | Positive margin from first measured layer | Both metrics, document-level p<10⁻³ |
| Five of six models | Euclidean competitor count | Hundreds over much of depth |
| All six models | Spearman, token rank vs cosine endpoint distance | 0.726 (Gemma-2B) to 0.982 (Gemma-7B) |
| Endpoint fits | Held-out diagnostics vs uniform sphere | 4.6–7.5x better |
| Qwen2.5-1.5B, layer 1 | Competitor fraction, same vs different final token | 0.364 vs 0.058 |
Three findings. Average preference starts almost immediately, but selectivity arrives late: the own endpoint beats the average alternative from the first measured layer in every model, while hundreds of individual competitors persist and only fall by orders of magnitude at depth (Gemma-7B shrinks earliest). Shrinkage is also not elimination from a fixed shortlist: cosine competitor sets gain members while losing them, most persistently in Qwen2.5-1.5B, which the straight-path theorem converts into proof of non-straight-line convergence. And Euclidean plateaus can coexist with continued directional gains, most visibly in the middle layers of Mistral-7B and Llama-3-8B.
Endpoint geometry tracks outputs. Tokens the query context ranks lower have endpoints farther away in cosine distance in all six models; the Euclidean version is significant in only four. Endpoint distance correlates with KL divergence between output distributions at Pearson 0.25 to 0.63.
Controls rule out trivial explanations. Dropping same-document alternatives barely moves the curves. Swapping in a same-final-token bank raises the competitor fraction (0.364 vs 0.058 at layer 1) while preference survives. Projecting out the input-token embedding direction leaves first-layer cosine preference intact in all six models (margins 0.0159 to 0.1124), though Euclidean preference is delayed to layers 2, 5, and 13 in three models. Early preference draws on both token identity and the broader context.
The fitted distributions transfer. The projected normal predicts cosine competitor curves with the lowest error in five of six models; a vMF mixture wins on Gemma-7B and places second elsewhere; the uniform sphere is worst in five.
For interpretability work, the endpoint-bank probe separates three quantities that layer-wise analyses tend to blur: distance to the endpoint, number of competitors, and concentration of the survivors. Decodable information and geometric discrimination are different properties, and this framework measures the second.
The straight-path theorem is the most reusable piece. A single observed entry into a competitor set certifies a non-straight trajectory using only per-layer activations, no weights, no probes. The finding that average preference is not selectivity also bears on early-exit and layer-pruning work that judges intermediate layers by decoding quality alone.
The honest framing: descriptive geometry, no interventions, no capability claims. What it delivers is a measurement framework and a falsifiable baseline, not a mechanism.
The authors' own list: the fitted distributions predict competition along observed trajectories but leave the origin and timing of residual updates unexplained; the functional role of geometric competition is open; competitor sets depend on the bank and metric, and reach zero at the endpoint by construction; endpoint geometry explains only part of output variation; turnover does not establish an internal search process. Interventions near margin crossings are proposed, not run.
Caveats from the read: 1,024 contexts of 256 tokens per model, one position per context, base models only up to 8B. The competitor notion depends on bank composition, so absolute counts shift with the sample. Inference ran at reduced precision, and the authors flag that entries near ties carry numerical caveats.