Stanford's TokenGate skims context tokens for up to 11.3x faster LLM prefill
CShorten30 · x · 2026-10-21
Researchers from Stanford and UC Berkeley (Liana Patel, Ion Stoica, Matei Zaharia, Carlos Guestrin, et al.) introduce Token-Gated Language Models: the model learns to skim context and conditionally forward only the tokens that matter, framing efficient LLMs' next frontier as conditional computation over context tokens — analogous to the dense-to-MoE shift, but along the context axis.
Key results (fine-tuned Qwen3-1.7B, accuracy-matched to standard prefill):
- TTFT: 425ms → 37ms (11.3x faster) on NIAH long-context retrieval; 286ms → 72ms (3.9x) on RepoBench code completion.
- Peak memory: 2.4GB → 0.7GB (3.4x lower) on NIAH; 1.6GB → 0.6GB (2.8x lower) on RepoBench.
A gate module at layer L keeps K tokens, with Token-Gated Attention pruning the forward pass in layers above. Code: github.com/token-gate-llms/TokenGate.
Related event: TokenGate: Learned Token Pruning Speeds Up LLM Prefill Up to 11x(3 posts)→
More from Infra
- Nscale listed 10,000MW of 'potential' capacity in its S-1 — built under 1% — SumitGup · 2026-10-22
- Musk says SpaceX will build 20GW of power and cooling next year — rohanpaul_ai · 2026-10-22
- Google and Meta open-source TorchTPU: native PyTorch backend for TPUs — PyTorch · 2026-10-22
- ExecuTorch now builds on Snapdragon X Elite laptops; two paths to on-device LLM inference — carrycooldude · 2026-10-22
- NVIDIA streams GTC Berlin After Hours: keynote reactions and developer outlook — NVIDIAAI · 2026-10-22
- Nvidia's SK Hynix deal locks up 37% of 2027 HBM, leaving ~15% for everyone else — Beth_Kindig · 2026-10-22