Why Only Two Layers Use Sliding Window Attention? Architecture Detail Sparks Debate
stochasticchasm · x · 2026-09-11
A technical discussion on a model that uses sliding window attention (SWA) on only its first two of 20 layers. The author speculates the prefill FLOPs savings seem modest but notes less global attention is cheaper, possibly echoing first-k-dense designs in MoEs. Cross-attention ablations likely existed, but letting the decoder attend directly to encoder hidden states within its own sequence — similar to MoDA's allow-KVs-from-earlier-layers approach — seems a good design choice.
More from Research
- RD-Forget: reversible, query-dependent forgetting for agent memory — MaryamMiradi · 2026-09-11
- Science Advances editor: no author has ever disclosed AI use despite policy — TuhinChakr · 2026-09-11
- YOCO explained: one shared KV cache reused across the model's second half — stochasticchasm · 2026-09-11
- PARSER: parallel chunk subagents with an RL-trained lead agent for long-context QA — omarsar0 · 2026-09-11
- Year-long study: heavier AI companion engagement predicts lower well-being — dhadfieldmenell · 2026-09-11
- DeepMind launches AlphaGenome Atlas, a 1TB navigable map of human DNA — neil_chilson · 2026-09-11