Latent Sparse Attention clusters KV values for 70x speedup over full attention
bendee983 · x · 2026-11-12
- Sparse attention eases LLMs' memory/compute bottleneck, but every new query still has to scan all stored KV values to find relevant ones.
- Latent Sparse Attention (LSA) restructures KV storage: values are organized into clusters (optionally hierarchical), so each query first compares against cluster centroids before doing fine-grained comparison only within relevant clusters.
- Result: over 20x faster than existing sparse attention methods and 70x+ faster than full attention on long-context tasks. An accompanying visualization walks through pooling keys, latent-space clustering, tree search, and top-k attention.
More from Infra
- Tesla deployed 46.7 GWh of battery storage in 2025, over 15% of the global total — XFreeze · 2026-11-12
- KQuant: open-source MLX quants cut Qwen KL divergence up to 3.7x vs public 4-bit — m4rkGS · 2026-11-12
- Baidu ships Kunlunxin M100 inference chip, M300 flagship slated for 2027 — Baidu_Inc · 2026-11-12
- e-means beats k-means in 1.4 epochs, delivering 10-45x GPU speedups on embeddings — pmttyji · 2026-11-12
- Five AWS re:Invent Predictions: Bedrock Agents and Trainium Take Center Stage — DavidLinthicum · 2026-11-12
- FlagOS × MiniCPM5-2B inference throughput challenge opens with ¥30K prizes — _jaydeepkarale · 2026-11-12