New Sparse Attention Method COBS

MatXComputing · x · 2026-07-15

This thread discusses the effectiveness of block sparse attention in long-context retrieval.

• The author notes that while hardware heavily favors block sparse attention, it remains rare in open-source LLMs.
• They observed that directly replacing dense attention with DeepSeek's NSA significantly degrades performance on synthetic retrieval tasks.
• On a 32k context benchmark, NSA scores only 0.300, compared to 0.904 for dense attention.
• The team proposes COBS (Cumulant Order Block Sparse Attention), claiming it boosts NSA's score from 0.300 to 0.820, closing about 86% of the gap.
• Additionally, COBS reduces KV-cache read traffic by 15.15x compared to dense attention, requiring only 1.21x more than the NSA baseline, while achieving lower position-dependent NLL.

The core conclusion is that the key to block sparse attention lies in block selection.

Original post →

More from Research

• Linear Digressions returns with a new season of audio essays on AI agents — ChrisGPotts · 2026-07-21
• ARISE study tested 45 AI clinical tools in 1,100 consult cases — HealthcareAIGuy · 2026-07-21
• Async OPD distillation doubles throughput while matching synchronous math accuracy — _lewtun · 2026-07-21
• A forecasting lesson on why R-squared alone led to overfitting and worse predictions — mdancho84 · 2026-07-21
• Google DeepMind’s Project Genie talk shows how creatives feed into model research — alexanderchen · 2026-07-21
• Nat Lambert says RL distillation does not use the strongest models as teachers — natolambert · 2026-07-21

Research channel →