Researchers Debate 1/d-Style Attention Scaling in Modern LLMs

A technical discussion on July 16 focused on whether attention in some modern LLM implementations may effectively follow a 1/d-style scaling, rather than the classic 1/sqrt(d_head) intuition. The topic matters because it touches the parameterization details of multi-head attention and could change how people interpret the behavior of current architectures. The posts are exploratory rather than definitive, and they do not present a single agreed explanation.

Key details

@stochasticchasm highlighted what they saw as a relatively new-looking observation: attention may exhibit 1/d scaling when QK normalization is computed per head. In the same vein, @SeunghyunSEO7 said that many LLMs now apply q_norm directly to q_proj(x), so each head’s q/k vectors are not necessarily scaled by sqrt(d_head) in the traditional way. In their discussion, the idea of 1/d_head scaling was presented as related to this implementation pattern.

Alternative explanations and early clues

@rishiiyer01 suggested another possible explanation: the 1/d behavior might be tied to a μP scaling factor, while also noting that per-head qk norm could still be part of the story. Separately, @SeunghyunSEO7 said that after taking inspiration from Alec’s architecture and adding head gain, activation gain, and embedding residual/skip connection, they observed performance improvements. No full experiment details were provided, so this serves only as a hint that per-head parameterization and gain design may affect outcomes.

What remains unresolved

Taken together, the cluster is best read as a research-level exchange about how modern attention scaling may depart from older textbook intuitions. The posts point to normalization and gain design as plausible drivers, but they do not establish a final mechanism or provide a comprehensive empirical comparison.

2026-07-16 ~ 2026-07-16 · 6 related posts

• [source] Observation on 1/d Attention Scaling — stochasticchasm · 2026-07-16
• Novelty of 1/d Attention Scaling Discussed — stochasticchasm · 2026-07-16
• [source] A New Explanation for 1/d Scaling in Attention — rishiiyer01 · 2026-07-16
• [source] Multi-Head Attention Scaling and Per-Head Gain — SeunghyunSEO7 · 2026-07-16
• Learnable Gain Parameters Enhance Model Performance — SeunghyunSEO7 · 2026-07-16

1 near-duplicate retellings: stochasticchasm