HiSA prunes 65-71% of Q·K computations in RoPE attention, boosting decoding throughput 41%
Promptmethus · x · 2026-11-23
A new arXiv paper proposes Hidden Sparse Attention (HiSA), arguing up to 70% of Q·K dot products in LLM attention are wasted. Unlike token-level sparsity methods, HiSA sparsifies the hidden feature dimension: exploiting RoPE's structure, it statically skips frequency-pair dot product terms based on token distance — high-frequency pairs matter less for distant tokens. The rule is parameter-free and needs no runtime selection.
Models trained from scratch with HiSA perform equal or better while computing only 1/3 of the query-key products. On NVIDIA H200 with 16K context and batch 128: decoding throughput rose from 9,200 to 13,000 tok/s (+41%), KV cache usage dropped from 45 to 29.1 GiB (-35%). The authors also show an existing 13B model can adapt with just 1.35B additional training tokens, extend HiSA to FlashAttention-3 kernels, and demonstrate dense attention naturally learns to suppress the components HiSA drops. Code is open-sourced.
More from Infra
- Koboldcpp ships v1.125.1 release — Fcking_Chuck · 2026-11-23
- Fireworks AI breach hits 150k Uber drivers; US tightens compute-access rules — DavidLinthicum · 2026-11-23
- Why cybercriminals skip AI attacks: too noisy, costly tokenomics, old TTPs still work — jcran · 2026-11-23
- Token consumption is shifting from dedicated GPU to usage-based inference — AccBalanced · 2026-11-23
- Microsoft publishes Azure AI architecture design guide with Foundry Agent reference architectures — adnan_hashmi · 2026-11-23
- Only 2% of announced 90GW self-powered data center capacity in US actually running — rohanpaul_ai · 2026-11-23