Tencent Hunyuan's SAS optimizes sparse attention end-to-end via gating inside attention softmax
_akhaliq · x · 2026-09-14
Tencent Hunyuan researchers propose SAS (Simple Attention Sparsification), which trains a context-ranking selector end-to-end with the language modeling loss by injecting continuous gating scores into attention logits. This avoids the gradient-blocking hard Top-K of prior methods and yields better context ranking under tight attention budgets, cutting quadratic cumulative attention cost. The paper details design choices crucial to making this simple approach work.
Related event: Tencent Open-Sources SAS Sparse Attention Method(4 posts)→
More from Research
- Researchers encode 200+ properties of SGLang inference server, uncover 10 bugs — LingmingZhang · 2026-10-31
- Expert panels map 21 clinical AI risks; adversarial tests show autonomous care still unsafe — AleksandraFaust · 2026-10-31
- Handwritten RAGEN code becomes the #3 hardest task in FrontierCS 2 — wzenus · 2026-10-31
- Fudan, Tencent Hunyuan and Tsinghua unveil ExplorationBench to test AI rule discovery — jiqizhixin · 2026-10-31
- Robotics benchmarks feel impossible; Drift builds evals for real deployment sites — jonstephens85 · 2026-10-31
- Researcher: capability transfer hits a ceiling when training against less human-like models — saurabh_shah2 · 2026-10-31