NoRA: a one-line normalization fix for LoRA boosts convergence, stability and fights forgetting
omarsar0 · x · 2026-09-07
Elvis Saravia highlights NoRA (Normalized Low-Rank Adaptation): a one-line change to LoRA that costs nothing and improves convergence, stability and forgetting.
Key insight: LoRA initializes the up-projection to zero, so early optimization is governed almost entirely by the down-projection — which tells you where to regularize.
Method: normalize the down-projection matrices during training; the authors also show normalizing once at initialization improves standard LoRA, the cheaper of the two options.
Results: benefits hold across pretraining, SFT and RL — faster convergence, better final performance, more stable training, less catastrophic forgetting. No added trainable parameters or inference-time compute, making it broadly applicable.
Related event: New Research Explains LoRA's Slow Convergence, Proposes One-Line NoRA Fix(3 posts)→
More from Research
- 31,352 repeated benchmark runs show LLM scores drift 3x more across days than within a day — ionutvi · 2026-09-07
- Vine robot grows around corners: dual-vine design steers 90° for surgery — Scobleizer · 2026-09-07
- Open-source pipeline makes fabricated citations structurally impossible, full walkthrough released — Waste_Public_2985 · 2026-09-07
- Sierra launches τ^τ-Bench: coding agents must build real customer-service agents, gaps vs experts are large — sierra-research · 2026-09-07
- Enoki unifies claim verification and hallucination localization, cutting resources while boosting accuracy — s-nlp · 2026-09-07
- Bandit Model Experiment Shows Users Lock Onto Familiar Options, Not the Best Ones — svk_roy · 2026-09-07