Thinking in Geometric Terms: What ReLU, LayerNorm, LoRA and VQ Do to the Data Space
techNmak · x · 2026-10-01
A useful ML habit: ask what a piece of math does to the space the data lives in. Many concepts have concrete geometric interpretations hidden behind equations and architecture diagrams.
- Feature maps: data that is awkward to separate in original coordinates can become linearly separable after transformation; the boundary is linear in the new representation but nonlinear viewed from the original one.
- ReLU networks: within a region where the active-ReLU pattern is fixed, a feed-forward network behaves as an affine function; move regions and the affine rule changes.
- LayerNorm: mean-centering restricts the vector to a hyperplane where coordinates sum to zero; dividing by standard deviation fixes the norm — in 3D the normalized vectors lie on a circle (plane ∩ sphere). Learned scale and bias free the output from that circle.
- LoRA: the weight update is a product of two small matrices, so its rank — and the subspace of output changes it can induce — is bounded by the chosen rank.
- Vector quantization: nearest-neighbour assignment replaces continuous representations with codebook entries, partitioning the space into Voronoi regions.
None of these is a variant of the others, but the same vocabulary — spaces, dimensions, regions, distances, rank, constraints — explains them all, often revealing more than the architecture diagram alone.
More from Research
- Research shows RL training breaks defenses against distillation attacks, evals give false security — terryyuezhuo · 2026-10-01
- True Positive Weekly #180: Xiaomi's MIT-licensed MiMo-V2.6, physicist-style LLM pruning, OpenHands — burkov · 2026-10-01
- Noam Brown: the only visitor to his poster is now his OpenAI colleague behind superhuman Stratego AI — polynoamial · 2026-10-01
- Berkeley Simons Institute opens fellowships for Fall 2027 diffusion generative modeling program — gautamcgoel · 2026-10-01
- A continuously running inner world as the experiential interface for a single AI persona — OrionForgeEcosystem · 2026-10-01
- HCOMP 2026 Best Paper Goes to Reader-Centered Summarization Evaluation Framework — windx0303 · 2026-10-01