Dev Ships Open-Source Sliding-Window Attention for HF LLMs, Hits 3.5MB KV Cache at 32K Context
ahsaor8 · reddit · 2026-09-06
The author turned their sliding-window attention (SWA) experiments into a reusable open-source project, swallm, for testing KV cache optimizations in long-context inference with HF causal LLMs.
- Implementation: A few attention sink tokens plus a bounded recent window, a ring-buffer KV cache, streaming/chunked prefill, and standard autoregressive decoding; SWA mode can be loaded from a pretrained model in a few lines of code.
- Qwen2.5-7B benchmarks (L40S): At 32K context, the KV cache is 1.84GB with full attention versus 3.5MB with SWA-64; at 64K, full attention OOMs while SWA stays bounded; decoding latency stays essentially constant as context grows.
- Positioning: The author doesn't claim SWA is universally better — the focus is the engineering trade-offs among context retention, KV memory, TTFT, and decoding speed; testing on Llama, Mistral, Gemma and reporting compatibility issues is welcome.
More from Infra
- Meta researcher: horrifying infra, not talent, is why OSS can't catch closed frontier — A_K_Nain · 2026-10-23
- Jane Street and NVIDIA speed up Mamba-2 with custom cuTile kernels — blelbach · 2026-10-23
- LatentLoop: asymmetric compute allocation cuts looped-MoE training cost by 19% — pmttyji · 2026-10-23
- Same AI request emits 8.5x more CO2 in Texas than in Sweden — jlundberg · 2026-10-23
- SK Hynix Pulls Plug on Solidigm US IPO as Chip Stocks Slide, Shares Down ~30% This Quarter — rohanpaul_ai · 2026-10-23
- Designer mocks Nvidia: buy idle GPUs or lose access to new ones—definitely not a bubble — michalmalewicz · 2026-10-23