15-year-old builds sliding window attention from scratch: 4.16x speedup, 21% less VRAM
vankownlaw · reddit · 2026-10-31
A 15-year-old developer implemented custom Sliding Window Attention from scratch to tackle the VRAM bottleneck of long-context inference on consumer GPUs. The approach physically truncates old tokens via chunking before computing the QxK matrix, avoiding massive NxN matrices and OOM errors.
Benchmark on a Tesla T4 (15,000 tokens, FP16):
- Standard PyTorch nn.MultiheadAttention: 867 ms, 1.12 GB VRAM
- Custom implementation: 208 ms, 0.89 GB VRAM
- Result: 4.16x faster, 21% less VRAM
The author acknowledges FlashAttention exists but built this to understand the mechanics; code isn't public yet. They're asking whether the approach is viable for video processing and what bottlenecks would appear at 100k+ tokens.
More from Infra
- Google's Project Suncatcher boots TPU in orbit, runs Gemma, beams back a haiku — OlivierBachem · 2026-10-31
- 5 cheap open models vote vs Opus: 89.6% accuracy at 1/10th the cost — PomegranateAble4339 · 2026-10-31
- MiniMax 3 video generation demoed running locally on a single RTX 3090 Ti — Impossible-Ad-3798 · 2026-10-31
- Microsoft offering 60,000 employees buyouts as it shifts spend from humans to datacenters — Polymarket · 2026-10-31
- Baseten set to raise $2B at up to $30B valuation as revenue triples — rohanpaul_ai · 2026-10-31
- Betelgeuse runs 68GB Qwen MoE bit-exact at 22 tok/s on RTX 3060 + 16GB RAM — zyxciss · 2026-10-31