Yang Songlin on how attention sink differs from gated attention: logZ and stacking
SonglinYang4 · x · 2026-11-13
Responding to a discussion linking attention sink and gated attention, the author adds two technical distinctions: attention sink cannot model elementwise gated attention, and gated attention is purely pointwise and agnostic to logZ, whereas attention sink takes logZ into account. He also notes gated attention is stackable.
More from Research
- Swabbing 96 sites: IIT Madras maps Chennai Metro's urban microbiome — ravi_iitm · 2026-11-13
- 38,101 ICLR 2027 Submissions Reveal an LLM Naming Crisis: 47 TRACEs, 42 PRISMs — zetalyrae · 2026-11-13
- Open-weights AI text detector ParaTrace v8 hits 99.2% TPR at 1% FPR, 5th on RAID leaderboard — AltruisticCouple3491 · 2026-11-13
- A curated 9-paper reading path into LLM quantization for beginners — East-Muffin-6472 · 2026-11-13
- HF open-sources Ettin rerankers: six sizes, SOTA, full training recipe — antoine_chaffin · 2026-11-13
- LightOn open-sources DenseOn: SOTA single and multi-vector retrieval models — antoine_chaffin · 2026-11-13