NoRA: a one-line normalization fix for LoRA boosts convergence, stability and fights forgetting
omarsar0 · x · 2026-09-07
Elvis Saravia highlights NoRA (Normalized Low-Rank Adaptation): a one-line change to LoRA that costs nothing and improves convergence, stability and forgetting.
Key insight: LoRA initializes the up-projection to zero, so early optimization is governed almost entirely by the down-projection — which tells you where to regularize.
Method: normalize the down-projection matrices during training; the authors also show normalizing once at initialization improves standard LoRA, the cheaper of the two options.
Results: benefits hold across pretraining, SFT and RL — faster convergence, better final performance, more stable training, less catastrophic forgetting. No added trainable parameters or inference-time compute, making it broadly applicable.
Related event: New Research Explains LoRA's Slow Convergence, Proposes One-Line NoRA Fix(3 posts)→
More from Research
- Nature Physics: reprogrammable refractive index enables arbitrary multimode wave control for ML — MarioKrenn6240 · 2026-10-27
- Wakaru open-sources ontologize: labeling datasets from model activations — amplifiedamp · 2026-10-27
- EMNLP Oral paper traces multilingual representations in LLMs with cross-layer transcoders — ZhijingJin · 2026-10-27
- GPT-6 Pro reruns crime research: GTA V's 2013 launch tied to violence dips in four regions — emollick · 2026-10-27
- OpenAI's Aaron Lou on leaving a Stanford PhD for OpenAI, diffusion LMs vs autoregression — aaron_lou · 2026-10-27
- CLT-Forge: Open-source toolkit for cross-layer transcoder training and attribution graphs — ZhijingJin · 2026-10-27