HiSA prunes 65-71% of Q·K computations in RoPE attention, boosting decoding throughput 41%

Promptmethus · x · 2026-11-23

A new arXiv paper proposes Hidden Sparse Attention (HiSA), arguing up to 70% of Q·K dot products in LLM attention are wasted. Unlike token-level sparsity methods, HiSA sparsifies the hidden feature dimension: exploiting RoPE's structure, it statically skips frequency-pair dot product terms based on token distance — high-frequency pairs matter less for distant tokens. The rule is parameter-free and needs no runtime selection.

Models trained from scratch with HiSA perform equal or better while computing only 1/3 of the query-key products. On NVIDIA H200 with 16K context and batch 128: decoding throughput rose from 9,200 to 13,000 tok/s (+41%), KV cache usage dropped from 45 to 29.1 GiB (-35%). The authors also show an existing 13B model can adapt with just 1.35B additional training tokens, extend HiSA to FlashAttention-3 kernels, and demonstrate dense attention naturally learns to suppress the components HiSA drops. Code is open-sourced.

Original post →

More from Infra

Infra channel →