NVIDIA lays out 8 design guidelines for sparse attention, reusing selections for 2.4x faster Top-k

NVIDIAAI · x · 2026-11-21

NVIDIA's technical blog details how to co-design sparse attention for efficient long-context inference: the indexer that scores all keys becomes the real bottleneck. Key points include running the indexer as a batched matmul with fused epilogue sized to one 200-row Tensor Core tile, using KH=1 with cross-head reduction to avoid linear decode growth, head dims of 128/256 with FP16 epilogue, cutting scored keys via compression/hierarchical indexing, fusing Top-k into the indexer kernel, and reusing the previous step's selection for 2.1x–2.4x faster Top-k. Companion resources: the Guess-Verify-Refine Top-k paper, fused Top-k in TensorRT LLM, and open-source kernels in FlashInfer.

Original post →

More from Infra

Infra channel →