Microsoft/Cornell paper: free pause tokens add compute at inference without extra context length

SonglinYang4 · x · 2026-09-16

dairai highlights a new paper from Microsoft and Cornell for anyone who wants thinking-token benefits but can't afford the context.

• Mechanism: a pause token buys the model extra compute for each next-token prediction, paid for with a sequence position. Free pause tokens deliver the same compute in a parallel prediction stream over a weight-shared backbone, riding an existing position instead of adding one.
• At inference it adds nothing to context length, leaves the KV cache unchanged, and costs essentially no extra latency, since extra FLOPs aren't the throughput bottleneck.
• On a 1B model it improves next-token prediction by 2-3 points.
• XLNet author Zihang Dai's collaborator remarks people keep reinventing ideas from XLNet (arXiv 1906.08237).

Original post →

More from Research

• Radial gate objective cuts diffusion transformer FID from 16.24 to 11.83 — Junha Park · 2026-11-02
• REA: async brain-cerebellum runtime for robots where policies never wait on the LLM — Pengfei Zhou · 2026-11-02
• Microsoft's MemSteer turns unlabeled user histories into agent memory supervision — MicrosoftResearch · 2026-11-02
• SciGame benchmark uses counterfactual worlds to expose LLM agents' priors vs evidence — Yusheng Zhao · 2026-11-02
• On Terence Tao's blog, Frank Calegari calls on mathematicians to adapt after OpenAI's 700-paper release — Hesamation · 2026-11-02
• Tsinghua & Microsoft's HarnessMM cuts agent token use by up to 87.93% while raising success rate — rohanpaul_ai · 2026-11-02

Research channel →