New Langford paper makes state-prediction separation nearly free with pause tokens
JohnCLangford · x · 2026-10-15
A new arXiv paper by John Langford and colleagues, "Almost Free State Prediction Separation," pushes state-prediction separation (SPS) to near-zero cost. Standard SPS splits the forward pass into a state stream and a prediction stream, relieving a model's hidden state of dual burdens, but costs 1.9x pretraining FLOPs. The paper introduces a free pause token — a prediction stream that writes no keys or values and rides existing positions — improving next-token prediction by 2-3 centinats on a 1B model with no added inference cost: no extra context length, no KV cache, no decode steps, and essentially no latency. Training overhead is minimized via a two-pass split keeping FlashAttention viable, a w=0 prediction window, a shared gated FFN, and phased training.
More from Research
- DuplexJev Cuts Autoregressive Decoding for Voice Agents, Answering 80 Speech Decisions in 0.1s — adventists-ai · 2026-10-15
- LLM-to-Flow Distillation Adapts Scientific Generators to Natural-Language Discovery Goals — rbhar90 · 2026-10-15
- ETH's Uncertainty-Aware World Model Makes Offline RL Work on Real Robots — breadli428 · 2026-10-15
- 167,356 simulated shop-days: paper shows eval world design writes the model behavior it scores — liminal_bardo · 2026-10-15
- Multilingual NLP roadmap: researcher maps open problems from data selection to LID across 350+ languages — StellaLisy · 2026-10-15
- Beyond RAG: Injecting Tensor Memory Cartridges Into Layer-3 of a Frozen LLM — Nearby_Indication474 · 2026-10-15