NVIDIA's SparDA Architecture Boosts Decoding Speed 1.7x for Long-Context LLMs
HankYeomans · x · 2026-08-13
NVIDIA researchers introduced SparDA (Sparse Decoupled Attention), a novel Transformer architecture designed to overcome bottlenecks in long-context LLM inference.
Key Improvements & Results:
- Achieves 1.7x faster decoding and improves long-reasoning accuracy by 6.5 points.
- Traditional sparse attention reduces compute but still suffers from KV cache growing with sequence length (causing PCIe transfer bottlenecks) and retains $O(T^2)$ complexity in the sparse selection step.
- SparDA introduces a fourth per-layer projection called Forecast alongside Q, K, and V. It predicts the KV blocks needed by the next layer, enabling lookahead selection that overlaps CPU-to-GPU prefetch with current-layer execution.
Engineering Details:
- Decoupled from the attention query, the GQA implementation uses one Forecast head per group, reducing selection overhead.
- Adds <0.5% parameters and requires training only the Forecast projections to match the original selector's attention distribution.
More from Infra
- Dev publicly asks neoclouds for discounts on a $30k GPU compute purchase — Sauers_ · 2026-10-16
- New hardware day: local LLM deployment the moment it arrives — lxfater · 2026-10-16
- Together AI's score centering fixes RL training collapse under quantized inference — nrehiew_ · 2026-10-16
- Inference is splitting prefill and decode across specialized chips, not just H100s — AccBalanced · 2026-10-16
- Rebasely launches in beta: Git hosting built for humans and coding agents — handotdev · 2026-10-16
- Drama3 TTS lands in audio.cpp: up to 5x real-time on RTX 5090, <8GB VRAM mode too — Acceptable-Cycle4645 · 2026-10-16