Stanford and UC Berkeley's TokenGate speeds long-context prefill by up to 11.3x
MundanePercentage674 · reddit · 2026-10-26
Researchers from Stanford and UC Berkeley introduce TokenGate (arXiv 2610.26754), a learned token-pruning method that lets LLMs 'skim' during long-context prefill.
- Lower layers process all N tokens; a learned gate keeps only top-K document tokens (question tokens are never pruned); upper layers run on K tokens only.
- Results: up to 11.3x faster time-to-first-token, 3.4x lower peak prefill memory, and 6.4x faster than the next-best pruner.
More from Infra
- Used RTX 3090s Now Top $1500 on eBay as Owner Hunts a Replacement — like_a_tensor · 2026-10-26
- Ternary 1.58-bit Nemotron 30B-A3B GGUF Runs 368K Context on 12GB GPU — pmttyji · 2026-10-26
- DeepSeek V4.1-Flash hits 49 tok/s decode on dual Radeon R9700 in local test — Longjumping-Elk-7756 · 2026-10-26
- Anthropic built Scion to replace git as AI-driven merges jumped from 2,500 to 30,000/day — bibryam · 2026-10-26
- NVIDIA shows RTX Spark powering sim-to-real robotics: Microduck robot learns somersault in simulation — Aiden_Tech_Ai · 2026-10-26
- MiniMax H3 community adds 8-step reference control, 4-step distill, Apple Silicon support — Hailuo_AI · 2026-10-26