Open-source sparse attention: Qwen 27B with no accuracy loss, 10M-token context
Immediate_Sky_6566 · reddit · 2026-11-03
A research team open-sourced a novel sparse attention method:
- Qwen 3 27B's attention can be sparsified with no observed accuracy loss
- Extending context to 10M tokens degrades some specific tasks, but most (like NIAH) remain fine
- Sparse attention can be natively trained on next-token prediction loss alone, matching full attention's validation loss while running faster
Everything is open: kernels, SGLang integration, paper and details. The team focuses on H200/B300-scale systems, but the method also applies to small open-weight models, making long context cheaper on local hardware.
Related event: Open-Source LSA Sparse Attention Extends Qwen 27B to 10M-Token Context(3 posts)→
More from Infra
- SF billboards slam cloud AI for stealing data; blogger predicts decade of on-device iPhone assistants — brandon_xyzw · 2026-11-03
- DeepSeek Held #1 on OpenRouter 23 of 25 Weeks, Now 32% of Tokens — teortaxesTex · 2026-11-03
- Running DeepSeek-V4.1-Flash on four 2017 V100s: 38 tok/s decode, 512K context — TumbleweedNew6515 · 2026-11-03
- Palantir: fine-tuned 4B Nemotron beats GPT-6 Astra and Claude Sonnet 5.5 at 1/200th the cost — eliano · 2026-11-03
- AI memory crunch kills the $100 phone: sub-$100 shipments down 60% — demian_ai · 2026-11-03
- Google Web AI Summit demo: 98% of a coding agent's tokens processed on-device with local Gemma — jason_mayes · 2026-11-03