Transformers Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs
Pavel Tikhonov · hf · 2026-09-25
This paper proposes the Superposition Linearity Hypothesis: when inputs from distinct text streams are linearly combined, an LLM outputs a superposition of the individual next-token distributions — the model can effectively hold two thoughts at once.
Key findings:
- Superposition linearity is an intrinsic property of the Transformer architecture rather than an emergent result of training; in fact, it tends to diminish as pretraining progresses.
- Lightweight fine-tuning substantially restores linearity, significantly reducing divergence between the predicted next-token distribution and the average of individual distributions.
- The authors introduce a guided decoding procedure that disentangles superposed outputs, enabling two coherent continuations to be generated simultaneously from a single forward pass.
The result has direct implications for mechanistic interpretability and parallel multi-stream generation.
Related event: Study shows LLMs can naturally hold two thoughts at once(2 posts)→
More from Research
- AI Futures Project moves up timeline: top-human AI researchers possible by 2027 as METR finds 1.9x progress acceleration — AccBalanced · 2026-11-11
- Fortnow asks Claude about complexity in the Aronson-Ambainis proof: Promise-BQP-complete — fortnow · 2026-11-11
- Backdooring AI agents by reprogramming fewer than 10 dormant neurons — no training needed — DimitrisPapail · 2026-11-11
- Duke and Princeton paper: 10 small parallel agent teams beat best-of-N and one giant team — rohanpaul_ai · 2026-11-11
- Deletion Tuning: Zeroing 0.05% of Weights Matches Full Fine-Tuning With Less Forgetting — Thrumpwart · 2026-11-11
- OpenTac: JHU/MIT open-source flex-PCB tactile skins that sensorize any robot surface — _krishna_murthy · 2026-11-11