Stratified Sparse Attention unifies retrieval and sampling to close the sparse-to-dense gap at extreme sparsity
Skylight1 · hf · 2026-10-23
A new Stratified Sparse Attention (SSA) method aims to reduce the gap between sparse and dense attention at extreme sparsity levels.
- Core idea: unify deterministic retrieval of important tokens with score-guided sampling, recovering attention mass that plain sparsification drops
- Cheap proxy score: an approximation of qᵀk/√d sets sampling probabilities; heavy-hitter tokens are always loaded while the remaining budget is filled by sampling with replacement
- Results: improved long-context inference accuracy and speed across multiple benchmarks
More from Research
- KIT formally verifies ~1,000 LangGraph functions, surfaces state-loss and auth bugs — LingmingZhang · 2026-10-23
- Researcher: vanilla LLMs/LRMs struggle with plan valuation and info-gathering in uncertain environments — xuanalogue · 2026-10-23
- MedSAM and EchoJEPA: Repurposing Meta's General AI Models for Medical Breakthroughs — BoWang87 · 2026-10-23
- Scott Wu's pub card game turned into a math RL environment built on OpenEnv — SergioPaniego · 2026-10-23
- AnyGoal research asks whether world models memorize trajectories instead of learning reusable rules — randall_balestr · 2026-10-23
- New paper with 25 coauthors examines feasibility of a hardwired pause on frontier AI training — wfithian · 2026-10-23