Transformers Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs
Pavel Tikhonov · hf · 2026-09-25
This paper proposes the Superposition Linearity Hypothesis: when inputs from distinct text streams are linearly combined, an LLM outputs a superposition of the individual next-token distributions — the model can effectively hold two thoughts at once.
Key findings:
- Superposition linearity is an intrinsic property of the Transformer architecture rather than an emergent result of training; in fact, it tends to diminish as pretraining progresses.
- Lightweight fine-tuning substantially restores linearity, significantly reducing divergence between the predicted next-token distribution and the average of individual distributions.
- The authors introduce a guided decoding procedure that disentangles superposed outputs, enabling two coherent continuations to be generated simultaneously from a single forward pass.
The result has direct implications for mechanistic interpretability and parallel multi-stream generation.
More from Research
- Theoretical Neuroscience podcast: how neuromorphic computing could speed up AI and cut energy use — neurovium · 2026-09-25
- Offloading Score, a new measure of AI over-reliance, earns NeurIPS 2026 Oral — Diyi_Yang · 2026-09-25
- AgenticGenTAMP: coding agents synthesize reusable robot TAMP policies across 28 envs — tomssilver · 2026-09-25
- Princeton-led work uses coding agents to synthesize reusable policies for generalized task and motion planning — tomssilver · 2026-09-25
- 98,000 simulated evaluations reveal robots making surprisingly clever physical reasoning — tomssilver · 2026-09-25
- RIKEN AIP team proposes practical Bayes-optimal fairness-accuracy tradeoff estimation with soft labels — RatnRajiv · 2026-09-25