Self-attention explained: why the √d_k scaling and dynamic weights matter

techNmak · x · 2026-09-21

A thorough engineering-oriented explainer of self-attention, walking from matrix mechanics to design rationale:

• Core form: learned projections Q=XWQ, K=XWK, V=XWV; each position's query is dot-producted with every key, scaled by √dk, then softmaxed per row.
• Why √dk: as dimensions grow, dot-product magnitudes push softmax into saturated, small-gradient regions; scaling keeps scores in a useful range.
• Q/K vs V roles: queries and keys determine weights; values are the vectors being mixed — output zi is a weighted combination of several value vectors, not a copied token.
• Weights aren't parameters: projection matrices are learned, but attention weights are computed from current activations and change with the input sequence.
• Causal masking: future positions are masked before softmax so their weights are zero.
• Multi-head attention: parallel heads with different projections, concatenated and passed through an output projection.

Everything reduces to Attention(Q,K,V) = softmax(QK^T/√dk)V: each position computes an input-dependent weighted combination of value vectors.

Original post →

More from Research

• EvoSkill v2 shows agents self-improving via persistent skills — and hacking their grader — 0xsachi · 2026-09-21
• Tencent open-sources T-Mem, an associative-recall memory architecture for long-term AI memory — aigclink · 2026-09-21
• Nature Computational Science review maps four roles for LLMs as human proxies, each needing its own validity criteria — iyadrahwan · 2026-09-21
• AI enters the Riemann Hypothesis arena: when machines prove what humans can't grasp — ugail · 2026-09-21
• GuardianAgent: EMNLP Paper Teaches AI Agents to Fight Back Against Web Tracking — flosalim · 2026-09-21
• ECDYSIS paper: patch agent harnesses by failure patterns, not failure counts — rohanpaul_ai · 2026-09-21

Research channel →