Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It
Zehao Jin, Ruixuan Deng, Junran Wang
cs.AI, cs.CL, cs.LG
2026-09-29
Freezing all weights and adding a rank-8 LoRA at one early layer lifts Qwen3-8B from 15.5% to 99% on 24-line reference chains; 13 base models natively follow only 1.4-3.6 lines.
Hand a language model a tiny program (K = apple, B = K, D = B, print(D)) and the answer should be apple. Pure in-context reference following, no world knowledge involved. Thirteen open-weight base models, spanning the Qwen3, Llama, OLMo-3, and Gemma-3 families plus the 292B MoE DeepSeek-V4-Flash, reliably follow only 1.4–3.6 lines of such chains (reach: the longest chain followed at 80%+ accuracy), median 2.2. Qwen3-8B scores 83.5% at three lines, 54% at four, 41.5% at five.
Both obvious fixes fail. Depth: OLMo-3 at 32 and 64 layers both sit near 2.6 lines, and DeepSeek-V4-Flash falls to 42% at six lines and 38% at eight, close to the 33% chance level. Recurrence: looped models reuse the same block at inference, yet Ouro-1.4B reaches 0, 1.6, 2.3, 2.2, and 2.2 lines across one to five loops, and Huginn stays at 1.4–1.6 after 8, 16, or 32 recurrences. Prior theory shows transformers can chase far longer pointer chains with logarithmic depth, so this is not a capacity ceiling. The weights hold computation that the default forward pass leaves unused.
The edit is about as small as possible. Every weight stays frozen; a rank-8 LoRA acts on the residual stream entering one early layer (h ← sh + BAh). That is 65,537 trainable parameters in Qwen3-8B, under 0.01% of the model, and the learned scalar s settles at 1.0006. Training combines answer cross-entropy with a KL penalty on WikiText text; perplexity moves only from 10.14 to 10.148. The LoRA acts on each token independently and moves no information between tokens, so all cross-token communication stays with frozen attention.
What it starts is a relay. By default, each program line carries its chain identity a couple of lines downstream and the computation stalls; the query resolves one or two more pointers and a late layer copies the value. With the LoRA, lines accumulate ancestor names and frozen heads read progressively further up the chain: two lines up in Qwen's layers 16–18, then three, four, six, and seven by layer 22, advancing within a narrow band of middle layers that repeats each loop. The causal evidence is strong: cutting each line's attention to its parent line in layers 14–22 returns six-, eight-, and twelve-line chains to chance (53%, 48%, 55%), while the same cut in layers 23–29 leaves 100%, 100%, 98%. When the Ouro LoRA acts only in the first loop, the unmodified later loops still carry the relay to about 25 lines.
Placement has a hard edge. Moving the Qwen3-8B LoRA from layer 20 to 21 collapses reach from 20.5 to 5.2 lines. A cutoff layer measured on the frozen model (where restoring a pointer's token stops recovering half its effect) predicted this limit within tolerance in three of four held-out models.
| Setting | Metric | Frozen | With LoRA |
| Qwen3-8B, 24-line chains | exact accuracy | 15.5% | 99% |
| Qwen3-8B longer-trained, 48-line | exact accuracy | 9% | 88% (reach 50 lines) |
| Ouro-1.4B, 8 loops | reach | 2.5 lines | ≥160 lines (87% at 160) |
| Huginn, 16 recurrences | reach | 1.4–1.6 lines | 58 lines |
| Qwen3-8B, fictional 2–5-hop facts | exact match | 41.2% | 97.4% |
The 160-line figure is four times the longest training chain, so the model extrapolates. Other interventions at the same layer work too: at layer 14, projection LoRA and FLAS flows all land near 90% on 24-line programs, while at layer 26 every one of them drops to 52–56%, coin-flip chain selection. On MuSiQue multi-hop QA, task-trained early LoRAs add 11.4, 9.4, and 17.9 exact-match points (Qwen3-8B, OLMo-3-7B, Llama-3.1-8B), and projection LoRA restricted to early layers keeps most of the all-layer gain (11.0 vs 12.8 in Qwen, 20.1 vs 20.9 in Llama).
For evaluation: a wrong direct answer does not establish that the weights cannot compose the references. Default behavior and computation accessible under a specified intervention are different quantities, and reporting only the former understates the model.
For adaptation, placement matters more than scale; a 65K-parameter edit at the right layer rivals full-layer tuning, and the useful window can be measured on the frozen model before any training. For looped and latent-reasoning models, repeating layers does not by itself lengthen the reasoning chain; a small learned start gives the loops something to do.
The authors list most of this themselves. The mechanistic evidence comes from synthetic in-context tasks; MuSiQue uses gold paragraphs and supports the placement claim, not the mechanism; detailed traces cover Qwen3-8B and Ouro-1.4B, and the two looped families are related. Each task format gets its own LoRA, so this is a demonstration that a tiny edit unlocks latent computation, not a universal fix, and no cross-task transfer of a single LoRA is shown. Placement prediction is modest: it missed OLMo-3-32B by five layers, and across nine models its 2.22-layer mean error essentially ties a post-hoc rule of 48% of depth (2.27).
Two more cautions sit in the numbers. The headline results use two-chain, level-ordered programs, the easiest configuration; interleaving cuts 24-line accuracy to 76.5% and a third chain cuts 16-line to 58%. Gains also reverse when loops run well past training counts, with Ouro degrading at twelve loops, consistent with overthinking. The best MuSiQue layers were chosen on the same development set, with no held-out split.