SPS Transformer Splits State and Prediction
burny_tech · x · 2026-07-15
A shared post highlighted a new Transformer study. The author argues that current Transformers use the same hidden state to simultaneously handle two competing responsibilities: maintaining state and predicting the next token.
This work proposes splitting these roles into two independent streams:
- One dedicated to state representation
- One dedicated to token prediction
The person sharing the post noted that this direction is reminiscent of Parallel Speculative Sampling (PaSS) from years ago. They praised the work for continuing to explore the decoupling of "state representation learning" from "token prediction learning." The original post also claimed that this SPS Transformer performs better than standard Transformers.
Related event: SPS Transformer: Separating State and Prediction into Dual Streams(5 posts)→
More from Research
- Navier-Stokes, Riemann, P vs NP: what this week's math buzzwords mean for you — koltregaskes · 2026-09-11
- Harry Collins: LLMs can't do frontier science because they can't invent new language — whoamisri · 2026-09-11
- The Waymo effect: how AI is quietly making research less collaborative — JohnHammersley · 2026-09-11
- Causal-only attention for non-generative tasks is wasteful, argues HF engineer — antoine_chaffin · 2026-09-11
- Catholic University of Chile researcher: scaling AI feedback is key to sustainable medical education — julianvarascom · 2026-09-11
- Nature paper images cellular activity across all organs, revealing body-wide circuits — arjunrajlab · 2026-09-11