New architecture's asymmetric QK-Norm with gamma but no V-Norm gamma draws attention
stochasticchasm · x · 2026-11-13
Researchers discuss architecture details shared by @SonglinYang4: the design keeps a learnable RMS gamma on Q/K norms but drops it on the V norm, and uses an unusual Q/K/V attention residual split — an asymmetric design choice commenters found notably interesting.
Related event: Researchers discuss novel asymmetric norm design in new model architecture(3 posts)→
More from Research
- Idea basins: why autonomous agents get stuck, and how frontier models explore differently — AlexGDimakis · 2026-11-13
- OpenAI trains GPT-6 with automated red-teaming, slash jailbreak success vs GPT-5.x — fouadmatin · 2026-11-13
- Speculative Self-Distillation Cuts Training Tokens 38–69% With Less Forgetting — heghbalz · 2026-11-13
- Inkling 2 Small's Attention Residual praised for cleaner depth transfer — simonguozirui · 2026-11-13
- Why reasoning distillation fails on 9B models: six LoRAs, mixed results — SignificantZebra5883 · 2026-11-13
- A smiley face drawn into a network's weights survives training untouched — danbri · 2026-11-13