Novelty of 1/d Attention Scaling Discussed
stochasticchasm · x · 2026-07-16
The post reviews a "very safe" release, considering it a solid foundational version suitable for further tweaking and secondary development. A subsequent reply adds that because qk norm is applied per head dimension, the use of 1/d scaling for attention appears to be quite novel.
Related event: Researchers Debate 1/d-Style Attention Scaling in Modern LLMs(6 posts)→
More from Research
- Structural ensembles beat single predictions in TCR:pMHC generalization study — quaidmorris · 2026-07-22
- Structural ensembles, not single predictions, drive robust TCR:pMHC generalization — quaidmorris · 2026-07-22
- A 3D ray plot shows how hard this Jacobian counterexample is to read — moultano · 2026-07-22
- LLM leaderboards are now often measuring the harness too, Gary Marcus warns — GaryMarcus · 2026-07-22
- New paper defines self-state attacks, showing OS defenses leave four agent-memory cases indistinguishable — Justgototheeffinmoon · 2026-07-22
- Krea 2 users recommend a two-pass Clownshark sampler setup for sharper image details — listopalafoto · 2026-07-22