LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
Wen Zan, Jiaqi Zhang, Jianchao Tan, Hong Liu, Cunguang Wang, Xiang Li, Duyue Ma, Guanyu Wu, Yifan Lu, Fengcun Li, Yerui Sun, Peng Pei, Yuchen Xie, Xunliang Cai
cs.AI, cs.CL, cs.DC, cs.LG
2026-08-03
Meituan's LongCat adds three indexing optimizations to DeepSeek sparse attention, matching full-attention quality at up to 3.6x faster 1M-context inference; an open 69B model ships with it.
Long context is a hard requirement for agentic coding and long-horizon tasks, but standard self-attention is O(L²) in sequence length. Sparse attention restricts each query to a small subset of key-value tokens; the catch is selecting that subset both accurately and cheaply. DeepSeek Sparse Attention (DSA) does this with a dedicated Lightning Indexer that scores every prefix token for each query, reaching near-lossless quality versus full attention. It is already in production in DeepSeek-V3.2 and GLM-5.
The Meituan team profiled DSA on their own accelerators and found two system-level bottlenecks behind that quality win. The first is discontinuous indexer output: token-level selection forces the core attention operator to gather KV vectors one at a time, and HBM bandwidth utilization collapses to about 4.5% of peak. The backward pass is worse, because scatteradd gradient writes conflict across cores. The second bottleneck is indexer overhead itself: the Lightning Indexer scores the entire prefix, so its aggregate cost is still O(L²), and at a 1024K context it accounts for 90% of per-layer latency. The component meant to save compute becomes the most expensive part of the layer.
LSA reshapes DSA with three complementary mechanisms, each aimed at one bottleneck.
Streaming-Aware Indexing (SI). The authors first measured where attention mass actually goes in a full-attention model. Attention sinks (a few initial tokens that absorb disproportionate weight because of softmax normalization) plus a local sliding window stably capture about 83% of attention mass at long range. Since that share is predictable, they fix it instead of letting the indexer select it dynamically. The total budget K=2048 is split into 16 sink tokens, 1024 sliding-window tokens, and 1024 dynamically selected sparse tokens. The fixed part lives in contiguous memory, so gathers become coalesced reads and HBM throughput recovers. They wrote a Hybrid Sparse Attention (HFA) kernel that runs the sliding-window and sparse branches on separate hardware streams and merges outputs with online-softmax rescaling.
Cross-Layer Indexing (CLI). Adjacent layers share 57.4% of their Top-K token sets on average, and reusing one layer's index on its neighbor still covers 93.2% of that neighbor's attention mass. They group consecutive layers in pairs (N=2): the first layer runs the indexer and the second reuses its result, halving indexing compute. The non-obvious step is cross-layer distillation, which trains the owner's indexer to predict the attention patterns of every layer in the group rather than only its own. Skipping it is costly: in the ablation, naive reuse without distillation drops needle-in-a-haystack accuracy at 128K from 96% to 70%. CLI also extends to the Multi-Token Prediction (MTP) steps used for speculative decoding, with all three steps sharing one index.
Hierarchical Indexing (HI). This one is training-free and plug-and-play at inference. It is a two-stage coarse-to-fine scheme: first recall the top 1024 candidate pages (page size 128) with cheap block-mean scoring, then run fine-grained token scoring only inside those pages. Per-query selection drops from O(L) to O(L/P + M·P). It only pays off at 256K and above; at shorter contexts the two-stage overhead exceeds the gain and it is slower than the flat indexer.
On the long-context benchmark HELMET, at the 69B-A3B scale (LongCat-Flash-Lite), LSA scores 59.02 versus 58.50 for full-attention MLA and 58.60 for standard DSA, effectively tied. General, reasoning, and coding benchmarks (MMLU, GPQA, AIME, HumanEval+, and others) show no consistent winner across the three.
| Attention | HELMET avg (69B-A3B) |
| Full MLA | 58.50 |
| Standard DSA | 58.60 |
| LSA | 59.02 |
Efficiency versus the DSA baseline:
| Comparison | Setting | Speedup |
| vs DSA | single-layer training @1024K | 1.92× fwd, 1.55× bwd |
| vs DSA | end-to-end prefill | 1.42–3.60× |
| vs DSA | end-to-end decode | 1.25–1.40× |
| vs MLA | single-layer training @1024K | 7.73× |
| HI alone | indexer @1024K | 4.11× |
Against full-attention MLA, single-layer training at 1024K is 7.73× faster, but below 64K LSA is slower than MLA; with their actual variable-length data mixture the crossover sits around 128K.
The contribution is turning sparse attention from quality-preserving-but-pricey into something cheap enough to deploy. DSA had already solved quality; LSA closes the deployment bill by amortizing indexer cost across layers (SI and CLI) and within a single pass (HI). The training efficiency is enough for Meituan to train LongCat-2.0 (1.6T-A48B) natively at up to one million tokens under a limited compute budget. They also open-source LongCat-Flash-Lite-Sparse (69B-A3B) with a native 1M context.
Two things matter most for practitioners. First, the recipe itself, a fixed sink-and-window budget plus cross-layer index sharing plus coarse-to-fine selection, is general engineering wisdom that should transfer to other sparse-attention designs, not just Meituan's architecture. Second, an open 1M-context 69B model is a directly usable artifact.
The caveat: every speedup here is measured on Meituan's in-house AI accelerators (domestic ASICs). The 4.5% bandwidth figure that motivates SI, and the gains it delivers, are hardware-dependent. NVIDIA GPUs have a different memory-access model, and the wins may not carry over one-to-one.
The authors state the main one plainly: LSA cuts attention compute but leaves the total KV-cache footprint intact, since every token still stores a KV entry. KV-cache partitioning and host-memory offloading relieve per-device pressure without lowering aggregate storage. Their suggested future direction is fusing LSA with orthogonal compression such as Cross-Layer Attention (CLA) or DeepSeek-V4's Compressed Sparse Attention (CSA).
Several other points are worth questioning. The 560B result where LSA beats MLA by 1.73 points is, by the authors' own admission, mostly because LSA produces shorter outputs and fewer responses get truncated, not a real capability gain. The open model's agentic jumps (SWE-Bench Verified 54→68, SWE-Bench Multilingual 38→59, τ²-Telecom 73→95) are not a clean attention ablation: the sparse version also got an updated long-context training corpus, so how much credit goes to LSA versus the data is unclear. CLI uses the conservative N=2, while the concurrent IndexCache holds quality at N=4 on a standard Transformer, which suggests Meituan's shortcut-connected architecture is more sensitive to index reuse and caps the achievable gain. And all quality conclusions rest on existing long-context benchmarks, which may not stress the indexing machinery hard enough to expose its real ceiling.