NVIDIA lays out 8 design guidelines for sparse attention, reusing selections for 2.4x faster Top-k
NVIDIAAI · x · 2026-11-21
NVIDIA's technical blog details how to co-design sparse attention for efficient long-context inference: the indexer that scores all keys becomes the real bottleneck. Key points include running the indexer as a batched matmul with fused epilogue sized to one 200-row Tensor Core tile, using KH=1 with cross-head reduction to avoid linear decode growth, head dims of 128/256 with FP16 epilogue, cutting scored keys via compression/hierarchical indexing, fusing Top-k into the indexer kernel, and reusing the previous step's selection for 2.1x–2.4x faster Top-k. Companion resources: the Guess-Verify-Refine Top-k paper, fused Top-k in TensorRT LLM, and open-source kernels in FlashInfer.
More from Infra
- Nebius says Q3 compute deals closed above $25M per megawatt, up from Q2 — Beth_Kindig · 2026-11-21
- llama.cpp adds llama serve: run Inkling 2 Mini GGUF locally with one command — simonguozirui · 2026-11-21
- TensorFold 1.3.2 ships 96 PRs: DeepSeek-V4.1 Metal decode 6-17% faster, 874 tok/s on Blackwell — AIFlow_ML · 2026-11-21
- Ollama CEO in Fortune: enterprise demand for open models keeps growing — ycombinator · 2026-11-21
- Nvidia plunges 13.17% in a single day to $126.75 — jfischoff · 2026-11-21
- vLLM ships Day-0 support for TML model, cuts GB200 decode latency to 3.84ms — vllm_project · 2026-11-21