CoWA and MALA attention: 7.4x faster forward pass at 128K, 28.5% fewer training FLOPs
BitExternal4608 · reddit · 2026-09-29
Two papers attack redundant attention compute. CoWindow Attention (CoWA) spreads distant context across KV heads with complementary windows whose union covers the full causal history — position-defined, no learned router. MassAlloc Attention (MALA) keeps full causal QK scoring but uses softmax statistics to skip low-contribution tile computation. On 8 H100s at 128K tokens: CoWA gives 7.4x forward / 8.6x backward / 3.0x decode speedups on the attention operator; MALA gives 2.2x/3.0x/1.6x. Scaled 0.6B→14B with continued training at 32B; at 14B/32K, total training FLOPs dropped 28.5% (CoWA) and 23.1% (MALA) with comparable capabilities. Authors note neither method is universally lossless vs dense attention. (arXiv:2609.32704, 2609.32712)
Related event: BAAI's CoWA Attention Speeds Up Long-Context Inference(2 posts)→
More from Infra
- Izwi 0.1.0 ships fully local voice AI with TTS, cloning and diarization — zinyando · 2026-10-26
- Frontier security models will need new compute classes, quantum likely first — BenBajarin · 2026-10-26
- DesignArena data: Gemini 3.8 Flash quality degrades sharply weeks after launch — legit_api · 2026-10-26
- Perplexity's 5B decision model runs free on your Mac, distilled from its 7B model — Saboo_Shubham_ · 2026-10-26
- Chart: Data centers funded via SPV structures keep hyperscaler debt off balance sheet — SumitGup · 2026-10-26
- OpenAI built an 'offline world' of cached web content to train frontier RL runs like GPT-7 — daniel_mac8 · 2026-10-26