Qwen gated attention paper shows attention sink is implicitly a sigmoid gate
miyashita_03 · x · 2026-11-13
X user miyashita03 notes that Inkling 2 Small uses both dynamic attention sink and gated attention, and suspects the former may be redundant since gated attention can already express the same functionality.
- In the quoted tweet he derives: rearranging the softmax, attention sink implicitly acts as a sigmoid gate, and gated attention is a more general version that learns element-wise gating per dimension
- This builds on the Qwen team's paper arXiv:2505.06708 (Gated Attention for LLMs): comparing 30 gating variants on 15B MoE and 1.7B dense models trained on 3.5T tokens, a head-specific sigmoid gate after SDPA consistently improves performance, training stability, and tolerance to larger learning rates
- Co-author Songlin Yang replies that the two mechanisms are "stackable"