Multi-Head Attention Scaling and Per-Head Gain
SeunghyunSEO7 · x · 2026-07-16
During a discussion on multi-head attention, someone pointed out that many LLMs currently apply qnorm directly to qproj(x), rather than scaling each head's q/k by \sqrt(dhead).
Replies mentioned that the 1/dhead scaling concept originates from muP. The reason for following a Law of Large Numbers (LLN) intuition is that q and k might already be aligned from the start. Another viewpoint suggested that if q and k are first scaled by a factor like \sqrt{dhead} gain, their dot product becomes dhead (qgain, kgain) cos(θ), meaning the "per head gain" design warrants further scrutiny.
Related event: Researchers Debate 1/d-Style Attention Scaling in Modern LLMs(6 posts)→
More from Research
- Structural ensembles beat single predictions in TCR:pMHC generalization study — quaidmorris · 2026-07-22
- Structural ensembles, not single predictions, drive robust TCR:pMHC generalization — quaidmorris · 2026-07-22
- enFoldX turns AlphaFold3 ensemble noise into a TCR–peptide–MHC predictor — quaidmorris · 2026-07-22
- enFoldX tops 8 neoantigen scans and an unseen-peptide benchmark — quaidmorris · 2026-07-22
- enFoldX reaches AUC 0.82 on human VDJdb and transfers to mouse at 0.76 — quaidmorris · 2026-07-22
- enFoldX extracts 106 interface and confidence features from AF3 ensembles — quaidmorris · 2026-07-22