Prefix Sliding paper: training-free 3x speedup for long reasoning via prefix + sliding window
cephaloform · x · 2026-10-14
A new arXiv paper, "Prefix Sliding for efficient test-time scaling," from Niklas Muennighoff, Percy Liang, Jason Wei, Andrew Ng, Luke Zettlemoyer, Yejin Choi, Mike Lewis and 12 others.
- Problem: full attention keeps entire reasoning traces in memory, making long-horizon test-time scaling prohibitively expensive
- Finding: most intermediate reasoning tokens lose importance as reasoning progresses
- Method: during reasoning, discard tokens outside the prefix (key instructions/tools) and a sliding window of the last few thousand tokens, capping memory regardless of reasoning length
- Results: training-free, it makes existing models 3x faster with no performance loss; RL training with Prefix Sliding scales to reasoning traces beyond 100k tokens with better performance
- Ablations show it beats summarizing intermediate tokens and vanilla sliding windows; code is open
A significant result for long-horizon agents and inference-cost work.
More from Infra
- A backyard "distributed data center" ADU startup for local LLM enthusiasts — dee_hw · 2026-10-14
- Dots Studio's DoSA speeds sparse attention indexer up to 2.74x over SGLang, open-sourced — dots-studio · 2026-10-14
- ShellElastic trains one supernet yielding all sizes, with provable hyperparameter transfer — Can Jin · 2026-10-14
- Report: Microsoft strains under data center spending and AI talent war, mulls cost cuts — SumitGup · 2026-10-14
- Pokémon Go players unwittingly built Niantic a 30-billion-image AI dataset — SumitGup · 2026-10-14
- Modal receives its first NVIDIA Vera CPU to power Sandboxes — charles_irl · 2026-10-14