NoRA: a one-line normalization fix for LoRA boosts convergence, stability and fights forgetting
omarsar0 · x · 2026-09-07
Elvis Saravia highlights NoRA (Normalized Low-Rank Adaptation): a one-line change to LoRA that costs nothing and improves convergence, stability and forgetting.
Key insight: LoRA initializes the up-projection to zero, so early optimization is governed almost entirely by the down-projection — which tells you where to regularize.
Method: normalize the down-projection matrices during training; the authors also show normalizing once at initialization improves standard LoRA, the cheaper of the two options.
Results: benefits hold across pretraining, SFT and RL — faster convergence, better final performance, more stable training, less catastrophic forgetting. No added trainable parameters or inference-time compute, making it broadly applicable.
Related event: New Research Explains LoRA's Slow Convergence, Proposes One-Line NoRA Fix(3 posts)→
More from Research
- New review: how prefrontal cortex dynamics turn world models into flexible behavior — AnnaCiaunica · 2026-10-27
- LIBERO-MAX: Robot Policies Drop 11–26 Points When the World Changes Mid-Task — drmapavone · 2026-10-27
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27