Q/K norm gets gamma but V norm doesn't: attention residual split draws praise
stochasticchasm · x · 2026-11-13
stochasticchasm comments on a model's training config: learnable gamma on the Q/K norm but not the V norm, and calls the q/k/v attention residual split a really cool design choice.
Related event: Researchers discuss novel asymmetric norm design in new model architecture(3 posts)→
More from Research
- Idea basins: why autonomous agents get stuck, and how frontier models explore differently — AlexGDimakis · 2026-11-13
- OpenAI trains GPT-6 with automated red-teaming, slash jailbreak success vs GPT-5.x — fouadmatin · 2026-11-13
- Inkling 2 Small's Attention Residual praised for cleaner depth transfer — simonguozirui · 2026-11-13
- Why reasoning distillation fails on 9B models: six LoRAs, mixed results — SignificantZebra5883 · 2026-11-13
- A smiley face drawn into a network's weights survives training untouched — danbri · 2026-11-13
- Open weights leak which GPU trained them — six Chinese models read as NVIDIA Blackwell — Antique_Atmosphere97 · 2026-11-13