SPS Transformer Separates State and Prediction
yoavartzi · x · 2026-07-14
A new paper proposes an interesting hypothesis about Transformers: a single hidden state simultaneously handles two competing roles: "saving state" and "predicting the next token."
The core finding suggests that splitting these responsibilities into two independent streams yields better performance than standard Transformers. The paper introduces the State-Prediction Separation (SPS) Transformer, which achieves significant pre-training gains through a very simple architectural modification.
Related event: SPS Transformer: Separating State and Prediction into Dual Streams(5 posts)→
More from Research
- Token Collapse: Study Finds Graph Soft Prompting Tokens in LLMs Fail to Encode Structure — Sapienza · 2026-10-27
- UniAction: unified action conditioning lets one world model handle mixed action types — ZeYanjie · 2026-10-27
- Infini-AI Lab open-sources Hermit, a framework for agent harness evolution and continual learning — BeidiChen · 2026-10-27
- Meta FAIR's Self-Direction turns agent failures into reusable LoRA skills, +13.8pp on Terminal-Bench — karen_ullrich · 2026-10-27
- DONIC hits 86.79% on bimanual dexterous tracking, crushing prior SOTA's 9.43% — liuziwei7 · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27