NoRA: Normalized LoRA Boosts Convergence and Stability
burny_tech · x · 2026-09-02
The paper NoRA: Normalized Low-Rank Adaptation introduces a simple yet effective method to stabilize LoRA's training dynamics.
Mechanism and Benefits:
- Since LoRA initializes the up-projection to zero, early optimization is governed by the down-projection. NoRA normalizes these down-projection matrices during training.
- The study shows that applying normalization only at initialization also improves standard LoRA, eliminating the need for repeated steps.
- Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and stability, and mitigates catastrophic forgetting.
- Requires no extra trainable parameters or inference-time computation.
Related event: NoRA: Normalized LoRA Improves Convergence and Stability(2 posts)→
More from Research
- Meme mocks academics stuck in 3-year journal queues as preprint authors get promoted — OdedRechavi · 2026-10-18
- TRACE Open-Sourced: Migrating G1 Controller to Full-Size Humanoid H2 in 8 Hours on One GPU — zhengyiluo · 2026-10-18
- Humanoid Robot Funding Tops $3B as Brian Potter Explains the VLA Models Driving Robots — scottleibrand · 2026-10-18
- Terence Tao on 'War Footing' as OpenAI Claims Navier-Stokes Solve Shakes Mathematics — AlexTensor · 2026-10-18
- Univ. of Tokyo paper: why your recorded voice feels eerie — a vocal uncanny valley — ssh4net · 2026-10-18
- MIT's SANDO Drone Planner Avoids Every Moving Obstacle Across 12 Real Flights — Olivier__OG · 2026-10-18