Decoupling State and Prediction in Transformers
burny_tech · x · 2026-07-14
Introduces the State-Prediction Separation Hypothesis: Transformers currently cram "next-token prediction" and "storing necessary future state" into the same hidden state, which forces the representation space to juggle competing objectives and hurts training efficiency.
The paper splits these roles into two streams: one for the persistent state of input tokens, and another for next-token prediction. The authors report:
- This approach yields lower loss across model scales from 53M to 1.7B parameters
- Downstream tasks see a 2-3 point improvement
- At 1.6B parameters, training with only 18B tokens matches the performance of a standard Transformer trained on 47B tokens
Related event: SPS Transformer: Separating State and Prediction into Dual Streams(5 posts)→
More from Research
- Token Collapse: Study Finds Graph Soft Prompting Tokens in LLMs Fail to Encode Structure — Sapienza · 2026-10-27
- UniAction: unified action conditioning lets one world model handle mixed action types — ZeYanjie · 2026-10-27
- Infini-AI Lab open-sources Hermit, a framework for agent harness evolution and continual learning — BeidiChen · 2026-10-27
- Meta FAIR's Self-Direction turns agent failures into reusable LoRA skills, +13.8pp on Terminal-Bench — karen_ullrich · 2026-10-27
- DONIC hits 86.79% on bimanual dexterous tracking, crushing prior SOTA's 9.43% — liuziwei7 · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27