Google researcher explains why 5:1 interleaved SWA won: XLA compile times and TPU constraints
teortaxesTex · x · 2026-10-17
Though framed as a snarky quote, the embedded exchange from Google researcher Aurko Roy contains real technical substance:
- Interleaved SWA with global layers at 5:1 and window size 512 evolved from merging Routing Transformer with scan layers — XLA prefers repeated blocks to cut compile time.
- Top-k sparse attention proved harder on TPUs; most successful sparsification happened on GPUs (DeepSeek, MiniMax, etc.).
- Follow-up discussion: at 5:1 with a 512 window, five of every six layers stop growing KV cache after 512 tokens — why these models fit 128k context on one box; GPU sparse variants still keep every token's KV for top-k selection, while SWA actually shrinks the cache.
The poster uses the exchange to comment that humans bring alpha while bots mostly parrot.
More from Infra
- Vercel's tsc-ts ports tsgo back to TypeScript, runs 2-4x faster than tsgo — cramforce · 2026-10-17
- Kevin O'Leary refiles Utah data center plan using 558M gallons of water a year — Zachly · 2026-10-17
- Wishing the RAM shortage makes school Chromebooks unaffordable — AI spillover hits education — RachelVT42 · 2026-10-17
- Compute Labs launches financing platform matching GPU operators with capital — paigeinsf · 2026-10-17
- Anthropic lands BB+ rating as Wall Street assembles $60B AI chip financing — basedjensen · 2026-10-17
- Qdrant tests Perplexity's PDF embeddings: 21× smaller vectors, 96.5% recall kept — jrxna · 2026-10-17