Tencent Hunyuan Open-Sources HiLS Sparse Attention Architecture

机器之心 · wechat · 2026-07-20

The Tencent Hunyuan team open-sourced HiLS-Attention, a novel hierarchical landmark sparse attention architecture. Through mathematical and structural innovations, it simultaneously resolves two major pain points of traditional sparse attention: insufficient expressiveness and non-differentiability end-to-end, breaking the dilemma of choosing between efficiency and performance. **Existing Pain Points**: Traditional chunked sparse attention often uses mean or max pooling to estimate block importance, which introduces systematic mathematical bias. Conversely, parameterized summary methods use discrete Top-K selection operations that cut off gradient backpropagation, preventing end-to-end optimization of the selection process. **HiLS Architectural Innovations**: - **Differentiable Scoring Function**: Utilizes first-order Taylor expansion to approximate block importance as "relevance term + entropy bias", adaptively unifying uniform and concentrated extreme distributions. - **Bridging Gradient Breakpoints**: Splits attention weights into intra-block and inter-block two-level Softmax, allowing proxy quality to participate directly in forward computation. This enables gradients from the language modeling Loss to directly guide the model in learning how to select blocks. **Experimental Results**: Validated on 345M to 7B models, short-text performance is nearly on par with full attention. Trained with only an 8K context, it achieves zero-shot extrapolation to 4M lengths. At a 512K context, prefill and decode are accelerated by 13.5x and 15.7x respectively. Furthermore, thanks to compression denoising effects, the model even outperforms full attention mechanisms on certain long-text retrieval tasks.

Original post →

More from Models

Models channel →