Why Only Two Layers Use Sliding Window Attention? Architecture Detail Sparks Debate
stochasticchasm · x · 2026-09-11
A technical discussion on a model that uses sliding window attention (SWA) on only its first two of 20 layers. The author speculates the prefill FLOPs savings seem modest but notes less global attention is cheaper, possibly echoing first-k-dense designs in MoEs. Cross-attention ablations likely existed, but letting the decoder attend directly to encoder hidden states within its own sequence — similar to MoDA's allow-KVs-from-earlier-layers approach — seems a good design choice.
Related event: Leaked DeepSeek V4.1 Flash Tech Report Reveals KV Cache-Obsessed Flagship(44 posts)→
More from Research
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- FLASH: GPU-native deformable sim learns cloth folding in minutes on one RTX 5090 — ChongZzZhang · 2026-10-26
- LEMON: zero-shot LLM misbehavior monitor flags hallucinations from attention maps — yoavartzi · 2026-10-26
- Behind the EMNLP keynote: LMs as simulated learners to generate child language acquisition hypotheses — najoungkim · 2026-10-26
- The paper behind the EMNLP 2026 keynote on bigram 'simulated learners' — najoungkim · 2026-10-26