Microsoft & Cornell Paper: Free Pause Tokens Boost Prediction With ~1.14x Training-Only Overhead
dair_ai · x · 2026-09-06
dairai highlights a Microsoft/Cornell paper on pause tokens: they buy the model extra compute per next-token prediction, paying only with a sequence position. Free pause tokens carry the same compute in a parallel prediction stream over a weight-shared backbone. At inference they add nothing to context length, leave the KV cache unchanged, and cost essentially no latency; a 1B model gains 2-3 centinats on next-token prediction. The cost moves to training — roughly 1.14x overhead versus an optimized pretraining pipeline. A notable alternative if thinking-token context costs are unaffordable.
Related event: Microsoft and Cornell Propose Free Pause Tokens for Better Predictions(2 posts)→
More from Infra
- llama.cpp team launches Llama: one-click local AI on your Mac menu bar — Hugging Face · 2026-10-23
- China's daily token use hits 500T, up 4x as average price doubles to ¥13.46 — teortaxesTex · 2026-10-23
- New all-reduce kernels deliver 12% decode speedup for DeepSeek-V4.1 on PCIe-connected GPUs — AlpinDale · 2026-10-23
- Offshore oil rigs could be converted into AI data centers powered by stranded gas — clemnt · 2026-10-23
- Same native UI, two stacks: Swift build is ~80MB Mac-only, Zig build is ~10MB cross-platform — jedisct1 · 2026-10-23
- SurfSense: an open-source local NotebookLM with MCP agents and a prompt-caching lesson — FurtiveMirth · 2026-10-23