No LoRA Learning Rate Can Match Full Fine-Tuning

thebasepoint found that LoRA ranks move in sync early in training and larger ranks act like higher effective learning rates; due to quadratic effects in W=AB, no LoRA learning rate can replicate full fine-tuning.

2026-11-07 ~ 2026-11-07 · 4 related posts

• LoRA ranks move in lockstep early in training; higher rank means higher effective LR — thebasepoint · 2026-11-07
• Why alpha shouldn't scale with rank: LoRA effective learning rate dynamics — thebasepoint · 2026-11-07
• Due to quadratic effects in W=AB, no LoRA learning rate emulates full fine-tuning — thebasepoint · 2026-11-07
• No LoRA learning rate can emulate full fine-tuning, finds agentic-experiments deep dive — thebasepoint · 2026-11-07