Qwen gated attention paper shows attention sink is implicitly a sigmoid gate

miyashita_03 · x · 2026-11-13

X user miyashita03 notes that Inkling 2 Small uses both dynamic attention sink and gated attention, and suspects the former may be redundant since gated attention can already express the same functionality.

Original post →