Dev Ships Open-Source Sliding-Window Attention for HF LLMs, Hits 3.5MB KV Cache at 32K Context
ahsaor8 · reddit · 2026-09-06
The author turned their sliding-window attention (SWA) experiments into a reusable open-source project, swallm, for testing KV cache optimizations in long-context inference with HF causal LLMs.
- Implementation: A few attention sink tokens plus a bounded recent window, a ring-buffer KV cache, streaming/chunked prefill, and standard autoregressive decoding; SWA mode can be loaded from a pretrained model in a few lines of code.
- Qwen2.5-7B benchmarks (L40S): At 32K context, the KV cache is 1.84GB with full attention versus 3.5MB with SWA-64; at 64K, full attention OOMs while SWA stays bounded; decoding latency stays essentially constant as context grows.
- Positioning: The author doesn't claim SWA is universally better — the focus is the engineering trade-offs among context retention, KV memory, TTFT, and decoding speed; testing on Llama, Mistral, Gemma and reporting compatibility issues is welcome.
Related event: swallm Brings Sliding Window Attention to HF LLM Inference(2 posts)→
More from Infra
- Grandma GPUs reborn: 2x Tesla P40 hits 48 tok/s on Qwen 27B via F16 cache + MTP — Jumpy-Operation-4615 · 2026-09-06
- Running H3 across a 3090 and unlocked 64GB CMP 170HX hits OOM in ComfyUI — JustinPooDough · 2026-09-06
- Baseten's Philip Kiely Launches Inference Engineering Book, Plus Learning Resources — kmeanskaran · 2026-09-06
- Polygres turns your Postgres into a hybrid search context layer for AI agents — Scobleizer · 2026-09-06
- TCS may invest up to $7.4 billion with TPG in a gigawatt AI campus in Hyderabad — emmanuelvivier · 2026-09-06
- KV cache pressure tool shows vLLM's advertised 2M-token cache can retain 3M after fixes — t4a8945 · 2026-09-06