JSSA: token-level sparse attention hits 6.43x decoding throughput at 128K context
pmttyji · reddit · 2026-10-29
A new paper proposes Joint Subspace Sparse Attention (JSSA), a token-level sparse attention method targeting the KV-cache traffic bottleneck in long-context LLM decoding.
- Selection quality: learns a shared low-dimensional query–key subspace by minimizing token-score preservation error; the non-convex objective is upper-bounded by a tractable marginal one with a closed-form spectral solution — direct refinement gave no measurable gain.
- Execution efficiency: a gather-free token-level indexed-attention kernel supports distinct per-KV-head selections under the standard GQA KV-cache layout.
- Results: across standard GQA, hybrid sliding-window/global attention, and MLA models, JSSA stays close to full-KV accuracy while avoiding the generation-length inflation and architecture-dependent failures of competing methods. On an NVIDIA B200 it reaches up to 6.43× full-KV decoding throughput at 128K context, matches model-native DSA reasoning accuracy without training, and retains 96–99% of DSA throughput at 700B scale.
Code: GitHub; paper: arXiv.
More from Infra
- "Insert coins to continue": meme laments the end of cheap AI tokens — Suspicious_Orchid770 · 2026-10-29
- Skip ComfyUI: run Flux-3-Image-Klein locally with stable-diffusion.cpp in four files — srigi · 2026-10-29
- GPU unlock is pointless for most users: quantized models, decode and gaming see zero speedup — Justify_87 · 2026-10-29
- US AI Power in Three Stages: 10GW+ Nuclear Contracts, Geothermal Live, SMRs Permitting — ingliguori · 2026-10-29
- Revolut self-hosted GLM: 60-70% faster delivery at a tenth of frontier API costs — AI Engineer · 2026-10-29
- Jarvislabs to Bring 1,024 NVIDIA B200 GPUs Online in December, Reservations Open — amaarora · 2026-10-29