New Research Explains LoRA's Slow Convergence, Proposes One-Line NoRA Fix

New research shows LoRA's slow convergence stems from an implicit low-rank preconditioner at step 0 that deviates from standard SGD. A one-line fix called NoRA normalizes LoRA's down-projection, improving convergence, stability, and catastrophic forgetting at zero cost.

2026-09-07 ~ 2026-09-07 · 3 related posts

• LoRA's first step isn't standard SGD: an implicit low-rank preconditioner throttles early fine-tuning — pbaylies · 2026-09-07
• New Research Tackles LoRA's Random Down-Projection Bottleneck for Faster Fine-Tuning — burkov · 2026-09-07
• NoRA: a one-line normalization fix for LoRA boosts convergence, stability and fights forgetting — omarsar0 · 2026-09-07