SPS Transformer Splits State and Prediction
burny_tech · x · 2026-07-15
A shared post highlighted a new Transformer study. The author argues that current Transformers use the same hidden state to simultaneously handle two competing responsibilities: maintaining state and predicting the next token.
This work proposes splitting these roles into two independent streams:
- One dedicated to state representation
- One dedicated to token prediction
The person sharing the post noted that this direction is reminiscent of Parallel Speculative Sampling (PaSS) from years ago. They praised the work for continuing to explore the decoupling of "state representation learning" from "token prediction learning." The original post also claimed that this SPS Transformer performs better than standard Transformers.
Related event: SPS Transformer: Separating State and Prediction into Dual Streams(5 posts)→
More from Research
- Token Collapse: Study Finds Graph Soft Prompting Tokens in LLMs Fail to Encode Structure — Sapienza · 2026-10-27
- The 'sweet lesson' is real: two kinds of ML problems need opposite strategies — mike64_t · 2026-10-27
- PixGR: Pixel-space generative decoding for fast feed-forward 3D reconstruction — kwangmoo_yi · 2026-10-27
- UniAction: unified action conditioning lets one world model handle mixed action types — ZeYanjie · 2026-10-27
- Infini-AI Lab open-sources Hermit, a framework for agent harness evolution and continual learning — BeidiChen · 2026-10-27
- Meta FAIR's Self-Direction turns agent failures into reusable LoRA skills, +13.8pp on Terminal-Bench — karen_ullrich · 2026-10-27