Latent Sparse Attention paper: indexing eats 98% of prefill at 10M tokens, LSA speeds one layer 22.7x
burny_tech · x · 2026-11-08
Prefrontal AI (with Torsten Hoefler and James Whittington among the authors) published Latent Sparse Attention (LSA), targeting attention cost at ultra-long context (1M+ tokens).
Problem: sparse attention saves compute, but finding the relevant tokens itself becomes extremely expensive at ultra-long context — sometimes 98% of attention's prefill time. Existing indexing is either still quadratic or loses accuracy at that scale.
Method: LSA hierarchically clusters keys in a learned latent space so queries score a hierarchy of clusters instead of every key, speeding up prefill, decode and training. Two variants:
- D-LSA: applicable to pre-trained models via distillation;
- N-LSA: trained natively on next-token loss alone.
Results: on GLM-5.3-FLASH (QWEN-3.8-27B) at 10M tokens, D-LSA speeds one attention layer 22.7x in prefill (72.4x vs full attention) and 4.6x in decode (42.5x vs full) with similar accuracy; N-LSA matches full-attention accuracy with a 1.6x per-layer training speedup at 128k tokens.
Related event: Open-Sourced Latent Sparse Attention Speeds Up Long Context 22.7x(2 posts)→
More from Infra
- Claude Opus migrates monitoring stack in a day, cutting Datadog costs 38x — BenBajarin · 2026-11-08
- Jinko Power plans 1GW solar-powered data center in Ningxia, a $361M project — teortaxesTex · 2026-11-08
- RTX 5090 stock drops to $6,759 at Newegg, down from $10k+ weeks — Beneficial-Ad-8127 · 2026-11-08
- 'GPU homelessness' on the rise: compute owners hunt for colo space with 10G and reliable power — markjeffrey · 2026-11-08
- In-flight Wi-Fi speedtest shows 122Mbps down, "best flight internet" yet — JoeJustice · 2026-11-08
- Data centers are scaling so fast that electricity and grid capacity are now the biggest bottleneck for superintelligence — XFreeze · 2026-11-08