Decoupling State and Prediction in Transformers
burny_tech · x · 2026-07-14
Introduces the State-Prediction Separation Hypothesis: Transformers currently cram "next-token prediction" and "storing necessary future state" into the same hidden state, which forces the representation space to juggle competing objectives and hurts training efficiency.
The paper splits these roles into two streams: one for the persistent state of input tokens, and another for next-token prediction. The authors report:
- This approach yields lower loss across model scales from 53M to 1.7B parameters
- Downstream tasks see a 2-3 point improvement
- At 1.6B parameters, training with only 18B tokens matches the performance of a standard Transformer trained on 47B tokens
Related event: SPS Transformer: Separating State and Prediction into Dual Streams(5 posts)→
More from Research
- OpenAI says long-horizon models need safety and alignment checks across full action sequences — rhiever · 2026-07-22
- A Reddit user proposes a consistency LoRA to keep anime and game scenes visually stable — ThirdWorldBoy21 · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- BlackboxNLP 2026 is recruiting extra reviewers after a high submission volume — hanjie_chen · 2026-07-22
- AWS shows self-distilled reasoning can preserve math and coding skills during SFT — AWS ML Blog · 2026-07-22
- UI2App shows screenshot fidelity still lags real interaction recovery — Grace Man Chen · 2026-07-22