TokenGate: learned token pruning speeds up LLM prefill up to 11.3x
Liana Patel · hf · 2026-10-21
Stanford and UC Berkeley researchers introduce TokenGate, a lightweight learnable token-pruning module that accelerates long-context prefill in transformers via token-gated attention.
- Unlike humans who skim, LLMs process every token at every layer; TokenGate is a drop-in component enabling end-to-end learnable pruning on existing trained models.
- To handle the non-differentiable top-K pruning operator, it borrows from MoE: token importance scores re-weight attention logits.
- Results: 1.4–11.3x faster time-to-first-token and 1.5–3.4x lower peak memory with no accuracy degradation; vs the best prior pruning method, up to 6.4x faster prefill and 2x lower peak memory.
- Open-sourced at github.com/tokengate-models/TokenGate.
More from Infra
- vLLM's specialized decode path hits 829 P90 TPS per user on four B300 GPUs, 26% faster — vllm_project · 2026-10-21
- vLLM says it can reproduce DeepSeek's inference performance, yielding $39/GPU/hr revenue — vllm_project · 2026-10-21
- vLLM matches DeepSeek's DeepInfer decode performance within 6%, generating $39/GPU/hr at off-peak pricing — vllm_project · 2026-10-21
- AMD brings EAGLE-4 speculative decoding to Instinct MI355X day 0, 1.6x faster than built-in MTP — hongyangzh · 2026-10-21
- Modal revives multiplayer game Curvytron to demo its new Sticky Sessions routing — charles_irl · 2026-10-21
- Seroter's reading list: TorchTPU open source, Claude Code's code review system — rseroter · 2026-10-21