Tencent Hunyuan Open-Sources HiLS Sparse Attention Architecture
机器之心 · wechat · 2026-07-20
The Tencent Hunyuan team open-sourced HiLS-Attention, a novel hierarchical landmark sparse attention architecture. Through mathematical and structural innovations, it simultaneously resolves two major pain points of traditional sparse attention: insufficient expressiveness and non-differentiability end-to-end, breaking the dilemma of choosing between efficiency and performance.
Existing Pain Points: Traditional chunked sparse attention often uses mean or max pooling to estimate block importance, which introduces systematic mathematical bias. Conversely, parameterized summary methods use discrete Top-K selection operations that cut off gradient backpropagation, preventing end-to-end optimization of the selection process.
HiLS Architectural Innovations:
- Differentiable Scoring Function: Utilizes first-order Taylor expansion to approximate block importance as "relevance term + entropy bias", adaptively unifying uniform and concentrated extreme distributions.
- Bridging Gradient Breakpoints: Splits attention weights into intra-block and inter-block two-level Softmax, allowing proxy quality to participate directly in forward computation. This enables gradients from the language modeling Loss to directly guide the model in learning how to select blocks.
Experimental Results: Validated on 345M to 7B models, short-text performance is nearly on par with full attention. Trained with only an 8K context, it achieves zero-shot extrapolation to 4M lengths. At a 512K context, prefill and decode are accelerated by 13.5x and 15.7x respectively. Furthermore, thanks to compression denoising effects, the model even outperforms full attention mechanisms on certain long-text retrieval tasks.
More from Models
- GPT-5.6 writes well but is instantly forgettable, user complains — BasedRaddka · 2026-09-11
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- awesome-llm-leaderboards: an open-source directory of LLM leaderboards, pricing tables, comparison tools — Last_Establishment_1 · 2026-09-11
- Anthropic claims it works to keep eval environments unidentifiable to models — MaxKannen · 2026-09-11
- Nex N2.5 Pro released on Hugging Face with 407GB of weights — jinnyjuice · 2026-09-11
- RoMa v2 image matching model unveiled in the usual black poster — ducha_aiki · 2026-09-11