Custom Attention Kernel Survives 32k Tokens Where Standard PyTorch OOMs — 11x Faster
vankownlaw · reddit · 2026-11-04
The author built a custom attention kernel for long-context that mimics human memory: early tokens are kept as "sinks" (core memories), with a rolling window and entropy decay for the rest. Benchmarked on a Tesla T4:
- 10k tokens: standard nn.MultiheadAttention 0.0893s vs custom kernel 0.0148s — 6.03x speedup
- 32k tokens: standard impl hits CUDA OOM (can't allocate the NxN matrix) while the kernel runs smoothly at 0.0148s (nominal 11.43x)
They note FlashAttention still scales quadratically in memory, whereas this approach physically never stores the full NxN matrix — and invite feedback on where to improve.
Related event: 15-year-old builds attention kernel: no OOM at 32k, 11x faster(2 posts)→
More from Research
- RSNA 2026 Deep Learning Lab to teach medical imaging VLM dataset building from scratch — PRouzrokh · 2026-11-04
- RSNA lab details: build medical VLM datasets with EHR cohorts, de-ID and QC in Colab — PRouzrokh · 2026-11-04
- ByteDance Seed's Recycler recycles hidden states for near-free latent reasoning — JohnCLangford · 2026-11-04
- Direct weight manipulation beats fine-tuning: pi0.5 hits 75% on cloth-folding — ZeeshanZiaML · 2026-11-04
- Researchers propose 'Four As' framework to untangle AI governance trade-offs — Brighter-Side-News · 2026-11-04
- Stanford's RoboArchitect: Coding Model Rewrites Robot Control Code for Under $150, Success Up to 43% — rohanpaul_ai · 2026-11-04