Implementing BatchNorm, LayerNorm, and GroupNorm from Scratch
jcflynnnn · reddit · 2026-08-01
To better understand the assumptions behind common normalization layers, a developer implemented BatchNorm, LayerNorm, and GroupNorm from scratch, comparing them on a simple 3-layer MLP trained on MNIST.
Key Findings:
- Dead Neurons Fixed: Without normalization, many neurons remain completely inactive across a batch. Normalization makes their outputs input-dependent.
- Equal Performance: On this task, all three methods performed similarly, boosting test accuracy from 84.1% to roughly 95-96%.
- Geometric Framing: LayerNorm projects samples onto a subspace where features sum to zero and fixes the norm. GroupNorm generalizes this concept to specific group structures.
The author shared the full code and visualizations, asking the community about practical scenarios where GroupNorm outperforms LayerNorm beyond small-batch vision tasks.
More from Research
- AgentIR: Retrievers That Read Agent Reasoning Hit 68% Accuracy — hllo_wrld · 2026-08-01
- LakeQuest Paper at COLM: A QA Benchmark for Messy, Real-World Data Lakes — hllo_wrld · 2026-08-01
- NeurIPS 2026 Announces Competition Track Covering Embodied AI and Science — NeurIPSConf · 2026-08-01
- Cross-Domain SFT: A New Framework for AI Alignment Research — ArthurConmy · 2026-08-01
- CG-World Releases 850k-Segment CG Dataset to Reshape World Models with Counterfactuals — TheTuringPost · 2026-08-01
- Jamming Voice AI Agents: Disrupting Responses via Signal Interference — plopesresearch · 2026-08-01