Sparse Attention via Indexer Distillation in Latent Space
A proposed sparse attention approach uses a block-level Indexer in compressed latent space, distilled from dense attention scores during training to produce block-sparse attention masks.
2026-08-27 ~ 2026-08-28 · 2 related posts
- Sparse Attention with Indexer: Distilling Dense Scores — nrehiew_ · 2026-08-27
- Technique: Sparse Attention via block-level indexers in latent space — SkyLi0n · 2026-08-28