NoRA: Normalized LoRA Boosts Convergence and Stability
burny_tech · x · 2026-09-02
The paper NoRA: Normalized Low-Rank Adaptation introduces a simple yet effective method to stabilize LoRA's training dynamics.
Mechanism and Benefits:
- Since LoRA initializes the up-projection to zero, early optimization is governed by the down-projection. NoRA normalizes these down-projection matrices during training.
- The study shows that applying normalization only at initialization also improves standard LoRA, eliminating the need for repeated steps.
- Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and stability, and mitigates catastrophic forgetting.
- Requires no extra trainable parameters or inference-time computation.
Related event: NoRA: Normalized LoRA Improves Convergence and Stability(2 posts)→
More from Research
- The Neural Basis of Laughter — burny_tech · 2026-09-02
- Percolation Theory: From Networks to ML & AI — burny_tech · 2026-09-02
- Exploring Parameter Spaces Where Small Mutations Evoke Varied Behaviors — ctjlewis · 2026-09-02
- Volunteering to review for AAAI with nothing in return — worth it? — OptimalOptimizer · 2026-09-02
- ‘Reverse Mathematics’ Illuminates Why Hard Problems Are Hard — burny_tech · 2026-09-02
- Research Reveals Generalization Phase Transitions in Diffusion Models — burny_tech · 2026-09-02