New model shows row norms ≈1 across all layers, echoing nGPT's hypersphere hypothesis
ChengleiSi · x · 2026-11-13
Ilya Loshchilov asks Yang Songlin whether sampled weight statistics — mean row norms of ≈1 across essentially all projection types — confirm the model places all vectors on a hypersphere as in nGPT.
The shared table shows attention Q/K/V, output, gate/sink, MoE router and unembedding projections all at ≈1 with relative spread of just 0.004–0.008%, while NVFP4-quantized expert gate/up (≈0.20%) and down (≈0.28%) projections show noticeably higher spread — suggesting an nGPT-style normalized architecture with FP4 expert layers as the main source of norm variance.
More from Research
- Study: AI startups raise 151% more in North America but no premium in Africa — danielrock · 2026-11-13
- MIT's AI copilot boosts structural buckling resistance 39% with only 15 seconds added — Dr_Singularity · 2026-11-13
- Robopedia launches Xperience 2, a 'gold standard' Physical AI dataset — liuziwei7 · 2026-11-13
- Zhejiang U & Baidu's Agent-R²: 4B agent beats larger open-source baselines on GDPval — ZhejiangUniversity · 2026-11-13
- BirdDreams driving world model generates perspective + BEV video from a single image — lealtaixe · 2026-11-13
- Ladish: AI takeover isn't magic, just scaling pretraining + RL further — JeffLadish · 2026-11-13