TokenGate: learnable token pruning cuts LLM prefill TTFT by up to 11.3x
pmttyji · reddit · 2026-10-21
The TokenGate paper introduces a drop-in module enabling end-to-end learnable token pruning for long-context LLMs via 'token-gated attention', an MoE-inspired reweighting of attention logits that sidesteps the non-differentiable top-K. It achieves 1.4–11.3x faster TTFT and 1.5–3.4x lower peak memory with no accuracy loss, beating prior pruning methods by up to 6.4x, and works on both standard (Qwen3) and hybrid-attention (Qwen3.5) models. Code is open source.
Related event: TokenGate: Learnable Token Pruning Speeds Up LLM Prefill up to 11x(2 posts)→
More from Infra
- embedl launches free edge device farm to test 1000+ optimized models on NVIDIA edge hardware — Course_Latter · 2026-10-21
- Taalas burns LLM into silicon: 8B model hits 17,000 tok/s on a single chip, 213x an RTX 5070 — IAM_274 · 2026-10-21
- NVIDIA launches DSX POD at GTC Berlin: pre-integrated AI factories deployable in 14 days — nvidia · 2026-10-21
- Jensen Huang shows off Vera Rubin node: assembly drops from 2 hours to under 5 minutes — rickasaurus · 2026-10-21
- PHP critical CVE scored 9.2 forces emergency patch: WordPress operators urged to prep before exploits hit — EthanJPerez · 2026-10-21
- Tenstorrent pitches 'one architecture at every size' from single card to supercluster — AIFlow_ML · 2026-10-21