Observation on 1/d Attention Scaling
stochasticchasm · x · 2026-07-16
The post makes an apparently "novel" observation: attention might exhibit 1/d scaling due to QK norm being computed per head.
This is a research-oriented technical judgment, contributing to the discussion on the scaling laws of attention mechanisms, rather than announcing a product or model release.
Related event: Researchers Debate 1/d-Style Attention Scaling in Modern LLMs(6 posts)→
More from Research
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- VidMap uses RoMa coarse matching on all frames, fine-scale only for keyframes — ducha_aiki · 2026-09-11
- Bug Hunt Bench author: leaderboard noise is about 2-3 points — PawelHuryn · 2026-09-11
- PNAS paper shows a tiny billiard-ball system is a universal computer — undecidability lives in two dimensions — eigensteve · 2026-09-11
- New paper: Absolute pose estimation from affine cues and gravity direction — ducha_aiki · 2026-09-11
- LoMa Paper Ships REALLY HardPairs Dataset, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11