Decoupling State and Prediction in Transformers
burny_tech · x · 2026-07-14
Introduces the State-Prediction Separation Hypothesis: Transformers currently cram "next-token prediction" and "storing necessary future state" into the same hidden state, which forces the representation space to juggle competing objectives and hurts training efficiency.
The paper splits these roles into two streams: one for the persistent state of input tokens, and another for next-token prediction. The authors report:
- This approach yields lower loss across model scales from 53M to 1.7B parameters
- Downstream tasks see a 2-3 point improvement
- At 1.6B parameters, training with only 18B tokens matches the performance of a standard Transformer trained on 47B tokens
Related event: SPS Transformer: Separating State and Prediction into Dual Streams(5 posts)→
More from Research
- Mathematician Daniel Litt Launches Problem Repo to Track Human vs AI Progress: 15 Problems, 1 Solved — littmath · 2026-09-11
- Open ECDSA.fail challenge uses AI agents to shrink Shor's-algorithm quantum circuits for Bitcoin keys — StefanoGogioso · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11
- Navier-Stokes, Riemann, P vs NP: what this week's math buzzwords mean for you — koltregaskes · 2026-09-11
- Fruit fly brain as an LLM: connectome-driven language model demo goes live — ngxson · 2026-09-11
- Harry Collins: LLMs can't do frontier science because they can't invent new language — whoamisri · 2026-09-11