François Fleuret: RMSNorm before and after projections is called Sandwich Norm
francoisfleuret · x · 2026-11-07
François Fleuret shares a TIL: applying RMSNorm both before and after the projections in a Transformer block is referred to as "Sandwich Norm" — a useful bit of architecture terminology for normalization layouts.
More from Research
- Michael Levin flags new book 'Open Questions in Developmental Biology' — drmichaellevin · 2026-11-07
- Google's AI detects hypertension from a regular smartwatch, beating in-office measurements — The Decoder · 2026-11-07
- Prompt scored higher? Model drift may be why — dev open-sources paired-testing tool — gael55x · 2026-11-07
- Pedro Domingos Shares an Explainer on How Transformers Work — pmddomingos · 2026-11-07
- Tsinghua's ParallelWorld uses test-time scaling for embodied agents, lifting success 56% to 83% — jiqizhixin · 2026-11-07
- RevealLayer ComfyUI node brings occlusion-aware image layer decomposition — TBG______ · 2026-11-07