A New Explanation for 1/d Scaling in Attention
rishiiyer01 · x · 2026-07-16
The post discusses a potential new explanation for using 1/d scaling in attention: it might be due to the mup scaling factor. The author adds that this could also be related to per-head qk norm.
Overall, it offers a technical hypothesis regarding an attention scaling mechanism, noting that this perspective "might be quite novel."
Related event: Researchers Debate 1/d-Style Attention Scaling in Modern LLMs(6 posts)→