Microsoft/Cornell paper: free pause tokens add compute at inference without extra context length

SonglinYang4 · x · 2026-09-16

dairai highlights a new paper from Microsoft and Cornell for anyone who wants thinking-token benefits but can't afford the context.

• Mechanism: a pause token buys the model extra compute for each next-token prediction, paid for with a sequence position. Free pause tokens deliver the same compute in a parallel prediction stream over a weight-shared backbone, riding an existing position instead of adding one.
• At inference it adds nothing to context length, leaves the KV cache unchanged, and costs essentially no extra latency, since extra FLOPs aren't the throughput bottleneck.
• On a 1B model it improves next-token prediction by 2-3 points.
• XLNet author Zihang Dai's collaborator remarks people keep reinventing ideas from XLNet (arXiv 1906.08237).

Original post →

More from Research

• Researcher: any agent acting over long horizons provably has a self-model and world model — chris_j_paxton · 2026-09-16
• Ben Antieau guest post on Terence Tao's blog: mathematics needs both 'fast math' and 'slow math' in the LLM era — littmath · 2026-09-16
• VisTW: a Traditional Chinese VLM benchmark for reading Taiwan — and an eval framework that caught a 36-point bug — piske_usagi · 2026-09-16
• Mind2Dialogue Simulates User Mental States to Train Human-Aware LLMs — Zixuan Wang · 2026-09-16
• NUS Survey Maps Six Roles for Foundation Models Across the Game Lifecycle — NationalUniversityofSingapore · 2026-09-16
• DCO: Only Update Direction Matters When Fine-Tuning Instruct Models — Fei Yuan · 2026-09-16

Research channel →